Skip to main content

Django Low-Level Cache API

Introduction​

In modern web development, performance optimization is crucial for delivering a smooth user experience. Django, a high-level Python web framework, provides a robust caching system to improve application performance. While Django offers a cache framework with high-level caching mechanisms like per-site, per-view, and template fragment caching, sometimes you need more granular control over what gets cached and how.

That's where Django's Low-Level Cache API comes in. This API gives you fine-grained control over caching specific Python objects, letting you decide exactly what to cache, for how long, and under what conditions. In this guide, we'll explore the Low-Level Cache API and learn how to leverage it effectively in your Django applications.

Understanding Django's Cache Framework​

Before diving into the Low-Level Cache API, let's briefly understand how Django's caching system works. Django's caching framework requires you to configure one or more caches in your settings file. Once configured, these caches are stored in a dictionary-like object called django.core.cache.caches.

Basic Cache Configuration​

First, let's set up a basic cache configuration in your settings.py file:

python
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'unique-snowflake',
    }
}

This configures a local memory cache as the default cache. Django supports several cache backends:

  • LocMemCache: Local memory cache (not suitable for production with multiple processes)
  • FileBasedCache: File-based cache
  • DatabaseCache: Database-based cache
  • MemcachedCache: Memcached-based cache
  • RedisCache: Redis-based cache (requires additional packages)

Accessing the Cache​

To use the Low-Level Cache API, you first need to get a reference to a cache object. There are two ways to do this:

Method 1: Using the default cache​

python
from django.core.cache import cache

# This uses the 'default' cache defined in CACHES

Method 2: Using a specific cache​

python
from django.core.cache import caches

# Access a specific cache
my_cache = caches['specific_cache_name']

Basic Cache Operations​

The Low-Level Cache API provides several methods for working with cached data:

1. Setting a Value in the Cache​

The most basic operation is storing a value in the cache:

python
# Set a value in the cache with the key 'my_key'
cache.set('my_key', 'my_value')

# Set a value with an expiration time (in seconds)
cache.set('another_key', 'another_value', 60 * 15)  # Cache for 15 minutes

2. Getting a Value from the Cache​

Retrieve a value from the cache using the get method:

python
# Get a value from the cache
value = cache.get('my_key')

# Get a value with a default if the key doesn't exist
value = cache.get('non_existent_key', 'default_value')

Example output:

>>> cache.set('name', 'Django')
>>> cache.get('name')
'Django'
>>> cache.get('unknown_key')
None
>>> cache.get('unknown_key', 'Default Value')
'Default Value'

3. Adding a Value (Only if the Key Doesn't Exist)​

If you want to set a value only if the key doesn't already exist in the cache:

python
# Returns True if the key was added successfully
success = cache.add('unique_key', 'unique_value')

Example usage:

python
>>> cache.add('new_key', 'new_value')  # Key doesn't exist yet
True
>>> cache.add('new_key', 'different_value')  # Key already exists
False
>>> cache.get('new_key')  # Original value remains
'new_value'

4. Checking if a Key Exists​

To check if a key exists in the cache without retrieving its value:

python
if cache.has_key('my_key'):
    # Do something if the key exists
    pass

5. Deleting a Key​

Remove a key from the cache:

python
cache.delete('my_key')

6. Increasing/Decreasing Values​

You can atomically increase or decrease numeric values in the cache:

python
# Increment a value by 1 (default)
cache.incr('counter')

# Increment by a specific amount
cache.incr('counter', 5)

# Decrement a value
cache.decr('counter')

# Decrement by a specific amount
cache.decr('counter', 3)

Example:

python
>>> cache.set('counter', 0)
>>> cache.incr('counter')
1
>>> cache.incr('counter', 10)
11
>>> cache.decr('counter')
10
>>> cache.decr('counter', 5)
5

7. Setting Multiple Keys at Once​

Set multiple key-value pairs in a single operation:

python
cache.set_many({
    'key1': 'value1',
    'key2': 'value2',
    'key3': 'value3'
}, timeout=30)  # Optional timeout in seconds

8. Getting Multiple Keys at Once​

Similarly, retrieve multiple keys in one operation:

python
values = cache.get_many(['key1', 'key2', 'key3'])
# values is a dictionary with found keys

Example:

python
>>> cache.set_many({'a': 1, 'b': 2, 'c': 3})
>>> cache.get_many(['a', 'b', 'c', 'd'])
{'a': 1, 'b': 2, 'c': 3}  # Note that 'd' is not in the result as it doesn't exist

9. Deleting Multiple Keys​

Delete multiple cache keys at once:

python
cache.delete_many(['key1', 'key2', 'key3'])

10. Clearing the Entire Cache​

Clear all keys from the cache:

python
cache.clear()

Practical Examples​

Let's see how the Low-Level Cache API can be applied in real-world scenarios:

Example 1: Caching Database Query Results​

One common use case is caching the results of expensive database queries:

python
def get_all_active_users():
    # Try to get data from cache first
    users = cache.get('active_users')
    
    if users is None:
        # Cache miss, query the database
        users = User.objects.filter(is_active=True).select_related('profile')
        
        # Store in cache for 1 hour
        cache.set('active_users', users, 60 * 60)
        
    return users

Example 2: Caching API Responses​

When working with external APIs, caching responses can dramatically improve performance:

python
import requests

def get_weather_data(city):
    cache_key = f'weather_{city}'
    
    # Try to get data from cache
    weather_data = cache.get(cache_key)
    
    if weather_data is None:
        # Cache miss, fetch from API
        response = requests.get(f'https://api.weather.com/current?city={city}')
        weather_data = response.json()
        
        # Store in cache for 30 minutes
        cache.set(cache_key, weather_data, 60 * 30)
    
    return weather_data

Example 3: Rate Limiting​

You can use the cache to implement basic rate limiting:

python
def can_make_request(user_id):
    cache_key = f'rate_limit_{user_id}'
    
    # Get the current request count
    request_count = cache.get(cache_key, 0)
    
    if request_count >= 100:  # Limit to 100 requests per hour
        return False
    
    # Increment the request count
    if request_count == 0:
        # First request, set initial value with expiry
        cache.set(cache_key, 1, 60 * 60)  # 1 hour expiry
    else:
        cache.incr(cache_key)
    
    return True

Example 4: Cache Versioning​

You can implement cache versioning to invalidate specific groups of cache entries:

python
def get_article_with_comments(article_id):
    # Get the current version for article comments
    version = cache.get('article_comments_version', 1)
    cache_key = f'article_{article_id}_comments_v{version}'
    
    data = cache.get(cache_key)
    
    if data is None:
        # Cache miss, get from database
        article = Article.objects.get(id=article_id)
        comments = article.comments.all()
        data = {
            'article': article,
            'comments': comments
        }
        cache.set(cache_key, data, 60 * 15)  # Cache for 15 minutes
    
    return data

def invalidate_article_comments():
    # Increment the version to invalidate all article comments caches
    version = cache.get('article_comments_version', 1)
    cache.set('article_comments_version', version + 1)

Cache Key Considerations​

When using the Low-Level Cache API, it's important to choose appropriate cache keys:

  1. Avoid Collisions: Ensure your keys don't conflict with other parts of your application
  2. Use Prefixes: Prefix keys with a module or function name
  3. Handle Special Characters: Be mindful that some cache backends have restrictions on key characters
  4. Keep Keys Short: Some backends have key length limitations

A good practice is to create a key generation function:

python
def make_cache_key(model_name, object_id, action=None):
    key = f'app_name:{model_name}:{object_id}'
    if action:
        key += f':{action}'
    return key

# Usage
key = make_cache_key('User', 123, 'profile')  # 'app_name:User:123:profile'

Cache Timeouts and Expiration​

When setting values in the cache, you can specify a timeout (in seconds) after which the value will expire:

python
# Cache for 5 minutes
cache.set('my_key', 'my_value', 60 * 5)

If you don't specify a timeout, Django uses the default timeout from your cache backend configuration. To set a key that never expires, use None as the timeout:

python
# Cache indefinitely
cache.set('permanent_key', 'permanent_value', None)

However, be careful with permanent cache entries as they may consume memory indefinitely and most cache backends don't guarantee permanent storage.

Handling Cache Failures​

Cache operations should never break your application. Always assume that caching might fail and provide fallback mechanisms:

python
def get_dashboard_data(user_id):
    try:
        # Try to get from cache
        data = cache.get(f'dashboard_{user_id}')
        if data is not None:
            return data
    except Exception as e:
        # Log the error but continue
        logger.error(f"Cache error: {e}")
    
    # Cache miss or error, generate the data from scratch
    data = generate_dashboard_data(user_id)
    
    try:
        # Try to cache the result
        cache.set(f'dashboard_{user_id}', data, 60 * 10)
    except Exception as e:
        # Just log the error and continue
        logger.error(f"Failed to set cache: {e}")
    
    return data

Summary​

Django's Low-Level Cache API provides powerful tools for fine-grained control over caching in your applications:

  • Set and retrieve values with cache.set() and cache.get()
  • Handle multiple keys with set_many(), get_many(), and delete_many()
  • Increment and decrement counters with incr() and decr()
  • Conditionally set values with add()
  • Clear the cache with clear()

By leveraging these methods, you can significantly improve your application's performance by reducing database queries, API calls, and other expensive operations.

Remember these best practices:

  • Use meaningful cache keys with prefixes to avoid collisions
  • Set appropriate timeouts to balance freshness and performance
  • Always provide fallbacks in case of cache misses or failures
  • Consider the limitations of your chosen cache backend

Additional Resources​

For further learning on Django's caching capabilities:

  1. Django's Official Documentation on Caching
  2. Memcached Documentation - if using Memcached as your backend
  3. Redis Documentation - if using Redis as your backend

Exercises​

  1. Implement a caching layer for a product catalog that caches individual products and category listings with different expiration times.

  2. Create a function that uses the Low-Level Cache API to implement a "recently viewed items" feature that remembers the last 5 items a user viewed.

  3. Build a simple analytics counter that uses incr() to track page views and ensures the counts are not lost if the application restarts.

  4. Implement a cache decorator that can be applied to any function to cache its results based on the function arguments.

With these tools and techniques, you're now well-equipped to implement efficient caching strategies in your Django applications!


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)