Nginx Logs
Introduction
Logs are a vital component of any web server, and Nginx is no exception. They provide a window into the server's operations, recording everything from user requests to server errors. Understanding Nginx logs is essential for debugging issues, monitoring server performance, and maintaining the health of your web applications.
In this tutorial, we'll explore how Nginx handles logging, the different types of logs it generates, how to configure them for your specific needs, and how to analyze them effectively.
Nginx Log Types
Nginx primarily uses two types of logs:
- Access Logs - Records information about each visitor request
- Error Logs - Records problems and errors encountered by the server
Let's examine each in detail.
Access Logs
Access logs record information about each request made to the server. By default, these logs are stored in /var/log/nginx/access.log
(on Linux systems).
Default Access Log Format
The default access log format in Nginx looks like this:
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
A sample log entry might appear as:
192.168.1.1 - - [14/Mar/2025:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326 "http://example.com/home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
Understanding Access Log Fields
Let's break down the components of this log entry:
$remote_addr
: Client IP address (192.168.1.1
)$remote_user
: Client identity if HTTP authentication is used (usually-
if not authenticated)$time_local
: Local server time when the request was received ([14/Mar/2025:13:55:36 +0000]
)$request
: The request line from the client ("GET /index.html HTTP/1.1"
)$status
: HTTP response status code (200
)$body_bytes_sent
: Number of bytes sent to the client (2326
)$http_referer
: The referring website ("http://example.com/home"
)$http_user_agent
: Client browser information ("Mozilla/5.0..."
)
Error Logs
Error logs record problems or errors Nginx encounters. By default, these logs are stored in /var/log/nginx/error.log
.
Default Error Log Format
A typical error log entry looks like:
2025/03/14 13:57:42 [error] 12345#0: *67 open() "/usr/share/nginx/html/missing-page.html" failed (2: No such file or directory), client: 192.168.1.1, server: example.com, request: "GET /missing-page.html HTTP/1.1", host: "example.com"
Understanding Error Log Fields
Let's break down this error log entry:
- Date and time:
2025/03/14 13:57:42
- Log level:
[error]
- Process ID:
12345#0
- Connection counter:
*67
- Error message:
open() "/usr/share/nginx/html/missing-page.html" failed (2: No such file or directory)
- Client IP:
client: 192.168.1.1
- Server name:
server: example.com
- Request:
request: "GET /missing-page.html HTTP/1.1"
- Host header:
host: "example.com"
Configuring Nginx Logs
You can customize Nginx logging behavior by modifying the configuration files.
Log Configuration Locations
Nginx log settings can be configured at different levels:
Each level inherits settings from its parent, but can override them as needed.
Configuring Access Logs
You can configure access logs in your Nginx configuration file:
http {
# Define a custom log format
log_format detailed '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time';
server {
# Use the custom format for this server
access_log /var/log/nginx/example_access.log detailed;
location /api/ {
# Disable access logs for API endpoints
access_log off;
}
}
}
Configuring Error Logs
Error logs can be configured like this:
# In the main context
error_log /var/log/nginx/main_error.log warn;
http {
# In the http context
error_log /var/log/nginx/http_error.log notice;
server {
# In a server context
error_log /var/log/nginx/example_error.log error;
}
}
Log Levels
Nginx error logs support different severity levels:
debug
- Most verbose, for debugging onlyinfo
- Informational messagesnotice
- Normal but significant conditionswarn
- Warning conditionserror
- Error conditionscrit
- Critical conditionsalert
- Action must be taken immediatelyemerg
- System is unusable
Each level includes all higher levels. For example, the error
level will include error
, crit
, alert
, and emerg
messages.
Log Rotation
To prevent log files from consuming all available disk space, we need to implement log rotation. This is typically handled by the logrotate
utility on Linux systems.
Sample logrotate Configuration
Create a file in /etc/logrotate.d/nginx
with the following content:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 nginx adm
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
endscript
}
This configuration:
- Rotates logs daily
- Keeps 14 days of logs
- Compresses old logs to save space
- Signals Nginx to reopen log files after rotation
Analyzing Nginx Logs
Now that we understand how to configure Nginx logs, let's explore how to analyze them effectively.
Basic Log Analysis with Common Unix Tools
- Count the number of requests:
wc -l /var/log/nginx/access.log
- Find the most frequently accessed pages:
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
- Count requests by HTTP status code:
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
- Find the most frequent visitors (IP addresses):
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
- Extract 404 (Not Found) errors:
grep ' 404 ' /var/log/nginx/access.log
Advanced Analysis with External Tools
For more comprehensive log analysis, several specialized tools are available:
- GoAccess - A real-time web log analyzer that runs in the terminal:
goaccess /var/log/nginx/access.log -c
-
ELK Stack (Elasticsearch, Logstash, Kibana) - A powerful set of tools for collecting, processing, and visualizing log data.
-
Nginx Amplify - Nginx's own monitoring tool that provides insights based on log analysis.
Real-World Example: Troubleshooting with Nginx Logs
Let's walk through a practical example of using Nginx logs to troubleshoot a common issue.
Scenario: Investigating 504 Gateway Timeout Errors
A user reports that they occasionally receive 504 Gateway Timeout errors when using your application.
Step 1: Find the relevant error logs
grep "504" /var/log/nginx/access.log
This might yield entries like:
192.168.1.10 - - [14/Mar/2025:14:30:22 +0000] "POST /api/upload HTTP/1.1" 504 167 "http://example.com/upload" "Mozilla/5.0" 60.123 60.000
Step 2: Check for patterns
Looking at the results, we notice:
- The errors only happen on
/api/upload
endpoint - The request time (
$request_time
) is consistently high (60+ seconds) - The upstream response time matches the request time
Step 3: Check error logs for more details
grep "/api/upload" /var/log/nginx/error.log
This might reveal:
2025/03/14 14:30:22 [error] 12345#0: *123 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.10, server: example.com, request: "POST /api/upload HTTP/1.1", upstream: "http://127.0.0.1:8080/api/upload", host: "example.com"
Step 4: Fix the issue
Based on our log analysis, the problem is that the upstream server (running on port 8080) is taking too long to respond. We could:
- Increase the timeout settings in Nginx:
location /api/upload {
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 300s; # Increase from default 60s to 300s
}
- Optimize the upload endpoint in our application to handle large files more efficiently.
Implementing Structured Logging with JSON
For more sophisticated environments, structured logging in JSON format can make log processing easier:
http {
log_format json_combined escape=json '{'
'"time_local":"$time_local",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"status": "$status",'
'"body_bytes_sent":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"http_referrer":"$http_referer",'
'"http_user_agent":"$http_user_agent"'
'}';
access_log /var/log/nginx/access.log json_combined;
}
This produces logs in a JSON format that can be easily parsed by log processing tools:
{"time_local":"14/Mar/2025:15:00:22 +0000","remote_addr":"192.168.1.1","remote_user":"-","request":"GET /index.html HTTP/1.1","status": "200","body_bytes_sent":"2326","request_time":"0.005","http_referrer":"http://example.com/home","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
Summary
Nginx logs are an invaluable resource for monitoring and troubleshooting your web server. In this tutorial, we've covered:
- The two primary types of Nginx logs: access logs and error logs
- How to understand the default log formats
- Configuring custom log formats and locations
- Log rotation to manage disk space
- Basic and advanced techniques for log analysis
- A practical troubleshooting example using log analysis
- Implementing structured logging with JSON
By mastering Nginx logs, you'll be better equipped to diagnose issues, monitor performance, and ensure the smooth operation of your web applications.
Additional Resources
Here are some exercises to reinforce your understanding of Nginx logs:
- Exercise 1: Configure a custom log format that includes response time and upstream response time.
- Exercise 2: Set up different logging configurations for different virtual hosts.
- Exercise 3: Write a simple bash script that extracts the top 10 slowest requests from your access logs.
- Exercise 4: Configure JSON-format logging and set up a basic ELK stack to visualize your logs.
Next Steps
Now that you understand Nginx logs, consider learning about:
- Nginx metrics and monitoring
- Security hardening with Nginx
- Advanced proxy and caching configurations
If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)