Skip to main content

Nginx Log Interpretation

Introduction

Nginx (pronounced "engine-x") is a popular web server known for its high performance, stability, reliability, and low resource consumption. When troubleshooting issues with Nginx, logs are your best friend. They provide valuable insights into what's happening with your server and can help identify problems quickly.

In this guide, we'll explore how to read and interpret Nginx logs effectively. You'll learn where logs are stored, what the different log formats mean, and how to analyze them to troubleshoot common issues. By the end, you'll be able to extract meaningful information from your Nginx logs to solve problems and optimize your server's performance.

Where to Find Nginx Logs

Nginx maintains two primary log files:

  1. Access Log: Records information about every request to your server
  2. Error Log: Records errors and diagnostic information

Default Log Locations

On most systems, Nginx logs are stored in one of these locations:

  • /var/log/nginx/
  • /usr/local/nginx/logs/
  • /usr/local/var/log/nginx/ (on macOS with Homebrew)

You can check your Nginx configuration to find the exact location:

bash
grep -r "access_log\|error_log" /etc/nginx/

Or check the main Nginx configuration file:

bash
nginx -T | grep "log"

Understanding the Access Log Format

The access log records information about each request that Nginx processes. By default, it uses the "combined" log format, which looks like this:

192.168.1.1 - user [10/Oct/2023:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 2326 "http://example.com/home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Let's break down this log entry:

ComponentExampleDescription
Remote Address192.168.1.1IP address of the client
Remote User-The hyphen indicates no authentication was required
UsernameuserUsername used for authentication (if any)
Timestamp[10/Oct/2023:13:55:36 -0700]Date, time, and timezone of the request
Request"GET /index.html HTTP/1.1"HTTP method, requested resource, and protocol
Status Code200HTTP response status code
Bytes Sent2326Size of the response in bytes
Referrer"http://example.com/home"URL that referred the client
User Agent"Mozilla/5.0..."Browser/client information

Common HTTP Status Codes

When troubleshooting, pay special attention to the status code:

  • 2xx: Success (e.g., 200 OK)
  • 3xx: Redirection (e.g., 301 Moved Permanently, 302 Found)
  • 4xx: Client Error (e.g., 404 Not Found, 403 Forbidden)
  • 5xx: Server Error (e.g., 500 Internal Server Error, 502 Bad Gateway)

Understanding the Error Log Format

The error log contains diagnostic information and records any errors that Nginx encounters. The format is usually:

2023/10/10 13:55:40 [error] 12345#0: *67890 open() "/usr/share/nginx/html/missing.html" failed (2: No such file or directory), client: 192.168.1.1, server: example.com, request: "GET /missing.html HTTP/1.1", host: "example.com"

Let's break down this error log entry:

ComponentExampleDescription
Timestamp2023/10/10 13:55:40Date and time of the error
Error Level[error]Severity level (debug, info, notice, warn, error, crit, alert, emerg)
Process ID12345#0Process ID and thread ID
Connection ID*67890Connection identifier
Error Messageopen() "/usr/share/nginx/html/missing.html" failed (2: No such file or directory)Detailed error description
Client IPclient: 192.168.1.1IP address of the client
Server Nameserver: example.comServer name from the configuration
Requestrequest: "GET /missing.html HTTP/1.1"HTTP request
Hosthost: "example.com"Host header from the request

Common Log Analysis Tasks

1. Checking for HTTP Errors

Find all 5xx server errors:

bash
grep " 5[0-9][0-9] " /var/log/nginx/access.log

Find all 404 errors:

bash
grep " 404 " /var/log/nginx/access.log

2. Finding High-Traffic Sources

Identify the top 10 IP addresses by request count:

bash
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 10

Find the most requested URLs:

bash
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 10

4. Monitoring Response Times (with log_format that includes $request_time)

If your Nginx is configured to log request times, you can find slow requests:

bash
awk '$NF > 1 {print $0}' /var/log/nginx/access.log  # Requests taking more than 1 second

5. Analyzing Traffic by Hour

See traffic distribution by hour:

bash
awk '{print substr($4, 14, 2)}' /var/log/nginx/access.log | sort | uniq -c

Real-World Troubleshooting Scenarios

Scenario 1: Diagnosing a 502 Bad Gateway Error

A 502 Bad Gateway error indicates that Nginx, acting as a proxy, received an invalid response from an upstream server.

Step 1: Check for 502 errors in the access log:

bash
grep " 502 " /var/log/nginx/access.log

Step 2: Look for related errors in the error log:

bash
grep "upstream" /var/log/nginx/error.log

Common Causes and Solutions:

  1. Upstream server is down:

    • Check if your application server (PHP-FPM, Node.js, etc.) is running
    • Restart the upstream service
  2. Timeout issues:

    • Increase timeouts in your Nginx configuration:
nginx
http {
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
fastcgi_connect_timeout 300s;
fastcgi_send_timeout 300s;
fastcgi_read_timeout 300s;
}
  1. Memory constraints:
    • Check for "worker process" or "connect() failed" messages in the error log
    • Increase memory limits for your application or adjust worker processes

Scenario 2: Investigating High Traffic Spikes

Sudden traffic increases can cause performance issues.

Step 1: Identify when the spike occurred:

bash
awk '{print substr($4, 2, 17)}' /var/log/nginx/access.log | sort | uniq -c

Step 2: Analyze requests during the spike period:

bash
grep "10/Oct/2023:13" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -nr

Step 3: Check for potential attackers:

bash
grep "10/Oct/2023:13" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -nr

Common Causes and Solutions:

  1. Legitimate traffic spike:

    • Scale your resources
    • Implement caching
  2. DDoS attack:

    • Implement rate limiting:
nginx
http {
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

server {
location /login {
limit_req zone=one burst=5;
}
}
}

Scenario 3: Debugging Slow Page Loads

Step 1: Identify slow-loading pages (if using custom log format with request_time):

bash
awk '$NF > 3 {print $7, $NF}' /var/log/nginx/access.log | sort -k2,2nr | head -n 10

Step 2: Check for related errors:

bash
grep "/slow-page.html" /var/log/nginx/error.log

Common Solutions:

  1. Enable caching:
nginx
http {
proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m;

server {
location / {
proxy_cache my_cache;
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
}
}
}
  1. Use gzip compression:
nginx
http {
gzip on;
gzip_types text/plain text/css application/javascript application/json;
gzip_min_length 1000;
}

Advanced Log Analysis Tools

For more complex analysis, several tools can help:

  1. GoAccess: Real-time web log analyzer that displays statistics in your terminal or as HTML reports.

Installation and usage:

bash
apt-get install goaccess
goaccess /var/log/nginx/access.log -c

Generate an HTML report:

bash
goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED
  1. ELK Stack (Elasticsearch, Logstash, Kibana): For large-scale log analysis and visualization.

  2. Filebeat: Lightweight log shipper to forward logs to a centralized location.

Customizing Nginx Log Formats

You can customize the log format to include additional information. Edit your Nginx configuration file:

nginx
http {
log_format detailed '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';

access_log /var/log/nginx/access.log detailed;
}

Common variables to include:

  • $request_time: Time to process the request (seconds)
  • $upstream_response_time: Time spent receiving the response from the upstream server
  • $pipe: "p" if the request was pipelined, "." otherwise
  • $gzip_ratio: Compression ratio
  • $http_host: Host header

Log Rotation

To prevent logs from consuming too much disk space, implement log rotation:

nginx
http {
access_log /var/log/nginx/access.log combined buffer=64k flush=5m;
}

Use logrotate (on most Linux distributions) by creating a file at /etc/logrotate.d/nginx:

/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 nginx adm
sharedscripts
postrotate
[ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
endscript
}

Summary

Understanding Nginx logs is crucial for effective troubleshooting and server optimization. By examining access and error logs, you can identify issues, monitor performance, and make informed decisions to improve your web server's reliability.

Key takeaways:

  • Access logs record all requests to your server
  • Error logs provide diagnostic information
  • HTTP status codes help categorize issues quickly
  • Common log analysis tasks can identify traffic patterns and errors
  • Custom log formats can include additional useful information
  • Log rotation prevents disk space issues

Exercises

  1. Find all 404 errors in your access log from the past 24 hours.
  2. Identify the top 5 IP addresses making requests to your server.
  3. Calculate the average response time for requests to a specific endpoint (if using a custom log format with response times).
  4. Set up a custom log format that includes the request processing time.
  5. Configure log rotation to compress logs older than one day.

Additional Resources



If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)