Nginx Log Interpretation

Introduction

Nginx (pronounced "engine-x") is a popular web server known for its high performance, stability, reliability, and low resource consumption. When troubleshooting issues with Nginx, logs are your best friend. They provide valuable insights into what's happening with your server and can help identify problems quickly.

In this guide, we'll explore how to read and interpret Nginx logs effectively. You'll learn where logs are stored, what the different log formats mean, and how to analyze them to troubleshoot common issues. By the end, you'll be able to extract meaningful information from your Nginx logs to solve problems and optimize your server's performance.

Where to Find Nginx Logs

Nginx maintains two primary log files:

1. Access Log: Records information about every request to your server
2. Error Log: Records errors and diagnostic information

Default Log Locations

On most systems, Nginx logs are stored in one of these locations:

• /var/log/nginx/
• /usr/local/nginx/logs/
• /usr/local/var/log/nginx/ (on macOS with Homebrew)

You can check your Nginx configuration to find the exact location:

bash

grep -r "access_log\|error_log" /etc/nginx/

Or check the main Nginx configuration file:

bash

nginx -T | grep "log"

Understanding the Access Log Format

The access log records information about each request that Nginx processes. By default, it uses the "combined" log format, which looks like this:

192.168.1.1 - user [10/Oct/2023:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 2326 "http://example.com/home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Let's break down this log entry:

Component	Example	Description
Remote Address	192.168.1.1	IP address of the client
Remote User	-	The hyphen indicates no authentication was required
Username	user	Username used for authentication (if any)
Timestamp	[10/Oct/2023:13:55:36 -0700]	Date, time, and timezone of the request
Request	"GET /index.html HTTP/1.1"	HTTP method, requested resource, and protocol
Status Code	200	HTTP response status code
Bytes Sent	2326	Size of the response in bytes
Referrer	"http://example.com/home"	URL that referred the client
User Agent	"Mozilla/5.0..."	Browser/client information

Common HTTP Status Codes

When troubleshooting, pay special attention to the status code:

• 2xx: Success (e.g., 200 OK)
• 3xx: Redirection (e.g., 301 Moved Permanently, 302 Found)
• 4xx: Client Error (e.g., 404 Not Found, 403 Forbidden)
• 5xx: Server Error (e.g., 500 Internal Server Error, 502 Bad Gateway)

Understanding the Error Log Format

The error log contains diagnostic information and records any errors that Nginx encounters. The format is usually:

2023/10/10 13:55:40 [error] 12345#0: *67890 open() "/usr/share/nginx/html/missing.html" failed (2: No such file or directory), client: 192.168.1.1, server: example.com, request: "GET /missing.html HTTP/1.1", host: "example.com"

Let's break down this error log entry:

Component	Example	Description
Timestamp	2023/10/10 13:55:40	Date and time of the error
Error Level	[error]	Severity level (debug, info, notice, warn, error, crit, alert, emerg)
Process ID	12345#0	Process ID and thread ID
Connection ID	*67890	Connection identifier
Error Message	open() "/usr/share/nginx/html/missing.html" failed (2: No such file or directory)	Detailed error description
Client IP	client: 192.168.1.1	IP address of the client
Server Name	server: example.com	Server name from the configuration
Request	request: "GET /missing.html HTTP/1.1"	HTTP request
Host	host: "example.com"	Host header from the request

Common Log Analysis Tasks

1. Checking for HTTP Errors

Find all 5xx server errors:

bash

grep " 5[0-9][0-9] " /var/log/nginx/access.log

Find all 404 errors:

bash

grep " 404 " /var/log/nginx/access.log

2. Finding High-Traffic Sources

Identify the top 10 IP addresses by request count:

bash

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 10

3. Identifying Popular Content

Find the most requested URLs:

bash

awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 10

4. Monitoring Response Times (with log_format that includes $request_time)

If your Nginx is configured to log request times, you can find slow requests:

bash

awk '$NF > 1 {print $0}' /var/log/nginx/access.log  # Requests taking more than 1 second

5. Analyzing Traffic by Hour

See traffic distribution by hour:

bash

awk '{print substr($4, 14, 2)}' /var/log/nginx/access.log | sort | uniq -c

Real-World Troubleshooting Scenarios

Scenario 1: Diagnosing a 502 Bad Gateway Error

A 502 Bad Gateway error indicates that Nginx, acting as a proxy, received an invalid response from an upstream server.

Step 1: Check for 502 errors in the access log:

bash

grep " 502 " /var/log/nginx/access.log

Step 2: Look for related errors in the error log:

bash

grep "upstream" /var/log/nginx/error.log

Common Causes and Solutions:

1. Upstream server is down:

   • Check if your application server (PHP-FPM, Node.js, etc.) is running
   • Restart the upstream service

2. Timeout issues:

   • Increase timeouts in your Nginx configuration:

nginx

http {
    proxy_connect_timeout 300s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;
    fastcgi_connect_timeout 300s;
    fastcgi_send_timeout 300s;
    fastcgi_read_timeout 300s;
}

3. Memory constraints:
   • Check for "worker process" or "connect() failed" messages in the error log
   • Increase memory limits for your application or adjust worker processes

Scenario 2: Investigating High Traffic Spikes

Sudden traffic increases can cause performance issues.

Step 1: Identify when the spike occurred:

bash

awk '{print substr($4, 2, 17)}' /var/log/nginx/access.log | sort | uniq -c

Step 2: Analyze requests during the spike period:

bash

grep "10/Oct/2023:13" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr

Step 3: Check for potential attackers:

bash

grep "10/Oct/2023:13" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -nr

Common Causes and Solutions:

1. Legitimate traffic spike:

   • Scale your resources
   • Implement caching

2. DDoS attack:

   • Implement rate limiting:

nginx

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
    
    server {
        location /login {
            limit_req zone=one burst=5;
        }
    }
}

Scenario 3: Debugging Slow Page Loads

Step 1: Identify slow-loading pages (if using custom log format with request_time):

bash

awk '$NF > 3 {print $7, $NF}' /var/log/nginx/access.log | sort -k2,2nr | head -n 10

Step 2: Check for related errors:

bash

grep "/slow-page.html" /var/log/nginx/error.log

Common Solutions:

1. Enable caching:

nginx

http {
    proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m;
    
    server {
        location / {
            proxy_cache my_cache;
            proxy_cache_valid 200 302 10m;
            proxy_cache_valid 404 1m;
        }
    }
}

2. Use gzip compression:

nginx

http {
    gzip on;
    gzip_types text/plain text/css application/javascript application/json;
    gzip_min_length 1000;
}

Advanced Log Analysis Tools

For more complex analysis, several tools can help:

1. GoAccess: Real-time web log analyzer that displays statistics in your terminal or as HTML reports.

Installation and usage:

bash

apt-get install goaccess
goaccess /var/log/nginx/access.log -c

Generate an HTML report:

bash

goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED

2. ELK Stack (Elasticsearch, Logstash, Kibana): For large-scale log analysis and visualization.

3. Filebeat: Lightweight log shipper to forward logs to a centralized location.

Customizing Nginx Log Formats

You can customize the log format to include additional information. Edit your Nginx configuration file:

nginx

http {
    log_format detailed '$remote_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent" '
                        '$request_time $upstream_response_time $pipe';
    
    access_log /var/log/nginx/access.log detailed;
}

Common variables to include:

• $request_time: Time to process the request (seconds)
• $upstream_response_time: Time spent receiving the response from the upstream server
• $pipe: "p" if the request was pipelined, "." otherwise
• $gzip_ratio: Compression ratio
• $http_host: Host header

Log Rotation

To prevent logs from consuming too much disk space, implement log rotation:

nginx

http {
    access_log /var/log/nginx/access.log combined buffer=64k flush=5m;
}

Use logrotate (on most Linux distributions) by creating a file at /etc/logrotate.d/nginx:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
    endscript
}

Summary

Understanding Nginx logs is crucial for effective troubleshooting and server optimization. By examining access and error logs, you can identify issues, monitor performance, and make informed decisions to improve your web server's reliability.

Key takeaways:

• Access logs record all requests to your server
• Error logs provide diagnostic information
• HTTP status codes help categorize issues quickly
• Common log analysis tasks can identify traffic patterns and errors
• Custom log formats can include additional useful information
• Log rotation prevents disk space issues

Exercises

1. Find all 404 errors in your access log from the past 24 hours.
2. Identify the top 5 IP addresses making requests to your server.
3. Calculate the average response time for requests to a specific endpoint (if using a custom log format with response times).
4. Set up a custom log format that includes the request processing time.
5. Configure log rotation to compress logs older than one day.

Additional Resources

• Nginx Documentation: Logging
• The Ultimate Guide to Logging
• Troubleshooting Nginx