Nginx Split Testing

Introduction

Split testing (also known as A/B testing) is a method used to compare two or more versions of a website or application to determine which one performs better. With Nginx, you can implement split testing by distributing traffic between different server versions without requiring any client-side code.

In this tutorial, you'll learn how to use Nginx's advanced features to implement split testing for your web applications. We'll cover various methods, from basic random distribution to more sophisticated cookie-based persistence.

Why Split Testing with Nginx?

Before diving into implementation, let's understand why Nginx is an excellent choice for split testing:

Performance: Nginx is designed for high performance and low resource consumption
Layer of Abstraction: Testing happens at the server level, requiring no changes to application code
Flexibility: Multiple methods available to distribute traffic
Scalability: Can handle high volumes of traffic while performing the split

Basic Split Testing Setup

Let's start with a simple 50/50 split between two versions of a website.

Prerequisites

Nginx installed on your server
Access to Nginx configuration files
Two different versions of your website/application deployed

Configuration Example

Here's a basic configuration that splits traffic randomly between two server groups:

nginx
http {
    # Define upstream servers for version A
    upstream version_a {
        server 10.0.0.1:8080;
        server 10.0.0.2:8080 backup;
    }
    
    # Define upstream servers for version B
    upstream version_b {
        server 10.0.0.3:8080;
        server 10.0.0.4:8080 backup;
    }
    
    # Split testing logic
    split_clients "${remote_addr}${http_user_agent}" $variant {
        50%     version_a;
        *       version_b;
    }
    
    server {
        listen 80;
        server_name example.com;
        
        location / {
            proxy_pass http://$variant;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_ip;
        }
    }
}

In this configuration:

We define two upstream server groups: version_a and version_b
The split_clients directive distributes traffic based on a hash of the client's IP address and user agent
50% of traffic goes to version A, while the remaining traffic goes to version B
We use a variable ($variant) to dynamically determine which upstream to route to

How It Works

The flow of a request through this configuration can be visualized:

Advanced Split Testing Techniques

Basic random splitting works for many cases, but sometimes you need more control. Let's explore some advanced techniques.

Weight-Based Distribution

Instead of an equal split, you might want to send different percentages of traffic to each version:

nginx
split_clients "${remote_addr}${http_user_agent}" $variant {
    20%     version_a;  # 20% of traffic
    *       version_b;  # 80% of traffic
}

This is useful when:

You're gradually rolling out a new version
You want to limit exposure to experimental features
You're testing with a small sample before full deployment

One limitation of the basic approach is that returning users might see a different version each time they visit. To ensure consistent experience, we can use cookies:

nginx
map $cookie_ab_test $preset_variant {
    "version_a"     version_a;
    "version_b"     version_b;
    default         "";
}

# Only for users without the cookie
split_clients "${remote_addr}${http_user_agent}" $split_variant {
    50%     version_a;
    *       version_b;
}

# Use preset if cookie exists, otherwise use split result
map $preset_variant $variant {
    ""      $split_variant;
    default $preset_variant;
}

server {
    listen 80;
    server_name example.com;
    
    location / {
        proxy_pass http://$variant;
        
        # Set the cookie to maintain consistency
        add_header Set-Cookie "ab_test=$variant; Path=/; Max-Age=3600";
        
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_ip;
    }
}

This configuration:

Checks if the user already has an ab_test cookie
If they do, it routes them to the same version they saw before
If not, it assigns them to a version based on the split logic
Sets a cookie so future visits maintain consistency

Feature-Based Testing

Sometimes you want to test specific features rather than entire versions. You can split traffic for specific URLs:

nginx
server {
    listen 80;
    server_name example.com;
    
    # Main site uses original version
    location / {
        proxy_pass http://original_version;
    }
    
    # New checkout process being tested
    location /checkout {
        split_clients "${remote_addr}" $checkout_variant {
            30%     new_checkout;
            *       original_checkout;
        }
        proxy_pass http://$checkout_variant;
    }
}

This allows you to test individual features while keeping the rest of the site consistent.

Collecting and Analyzing Test Data

Split testing is only valuable if you can measure results. Here's how to send test data to your analytics system:

nginx
server {
    listen 80;
    server_name example.com;
    
    location / {
        proxy_pass http://$variant;
        
        # Add custom header for analytics
        proxy_set_header X-AB-Variant $variant;
        
        # Log the variant for server-side analysis
        access_log /var/log/nginx/ab_test.log combined_ab;
    }
}

# Custom log format with variant information
log_format combined_ab '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" '
                      '"$variant"';

With this configuration:

Each request includes which version was shown (in the X-AB-Variant header)
Server logs capture this information for analysis
You can pass this data to analytics tools to compare metrics between versions

Real-World Example: Testing a New UI

Let's walk through a complete example of testing a new UI design:

nginx
http {
    upstream original_ui {
        server 10.0.0.1:8080;
        server 10.0.0.2:8080;
    }
    
    upstream new_ui {
        server 10.0.0.3:8080;
        server 10.0.0.4:8080;
    }
    
    # Create variant mapping based on cookie or split
    map $cookie_ui_test $preset_variant {
        "original"  original_ui;
        "new"       new_ui;
        default     "";
    }
    
    split_clients "${remote_addr}${http_user_agent}" $split_variant {
        25%     new_ui;
        *       original_ui;
    }
    
    map $preset_variant $ui_variant {
        ""      $split_variant;
        default $preset_variant;
    }
    
    server {
        listen 80;
        server_name example.com;
        
        location / {
            # Route to the appropriate version
            proxy_pass http://$ui_variant;
            
            # Set cookie for consistency (30 days)
            add_header Set-Cookie "ui_test=$ui_variant; Path=/; Max-Age=2592000";
            
            # Add variant to response headers for client-side tracking
            add_header X-UI-Variant $ui_variant;
            
            # Standard proxy headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
        
        # Admin endpoint to force a specific version (for testing)
        location /set-version {
            if ($arg_version = "original") {
                add_header Set-Cookie "ui_test=original_ui; Path=/; Max-Age=2592000";
                add_header Content-Type text/plain;
                return 200 "Version set to original";
            }
            
            if ($arg_version = "new") {
                add_header Set-Cookie "ui_test=new_ui; Path=/; Max-Age=2592000";
                add_header Content-Type text/plain;
                return 200 "Version set to new";
            }
            
            return 400 "Invalid version specified";
        }
    }
}

This comprehensive example includes:

A 25/75 split between new and original UI
Cookie persistence for 30 days
Headers for analytics integration
An admin endpoint to force a specific version for testing

Common Challenges and Solutions

Challenge: Users Seeing Inconsistent Experiences

Solution: Use cookie-based persistence as shown in the examples above.

Challenge: Results Contamination

Problem: Analytics data getting mixed between versions.

Solution:

nginx
# Add version tracking to all internal requests
proxy_set_header X-AB-Variant $variant;

# Separate logs for each variant
map $variant $variant_log {
    version_a   /var/log/nginx/version_a.log;
    version_b   /var/log/nginx/version_b.log;
}

access_log $variant_log;

Challenge: Caching Issues

Problem: Cached responses might not respect the split.

Solution:

nginx
# Include variant in cache key
proxy_cache_key "$scheme$host$request_uri$variant";

Summary

Nginx provides powerful tools for implementing split testing that can help you make data-driven decisions about your website or application. In this guide, we've covered:

Basic random traffic splitting
Weight-based distribution
Cookie-based persistence for consistent user experience
Feature-based testing
Analytics integration
A comprehensive real-world example

By implementing split testing with Nginx, you can test changes safely, gather valuable data, and improve your application based on user behavior rather than assumptions.

Additional Resources

For further exploration of Nginx split testing capabilities:

The official Nginx Split Clients Module documentation
Learn more about Nginx load balancing
Explore the Nginx Proxy Module for additional configuration options

Exercises

To reinforce your understanding:

Set up a basic 50/50 split test between two versions of a simple web page
Modify the configuration to use a 70/30 split with cookie persistence
Implement a feature-based test that only splits traffic for a specific URL path
Add custom logging to track which version each user sees
Create a configuration that allows gradually increasing traffic to a new version over time

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction​

Why Split Testing with Nginx?​

Basic Split Testing Setup​

Prerequisites​

Configuration Example​

How It Works​

Advanced Split Testing Techniques​

Weight-Based Distribution​

Cookie-Based Persistence​

Feature-Based Testing​

Collecting and Analyzing Test Data​

Real-World Example: Testing a New UI​

Common Challenges and Solutions​

Challenge: Users Seeing Inconsistent Experiences​

Challenge: Results Contamination​

Challenge: Caching Issues​

Summary​

Additional Resources​

Exercises​