Skip to main content

Security and Authentication in Prometheus

Introduction​

Security is a critical aspect of any production monitoring system. Prometheus, as a powerful monitoring and alerting toolkit, collects and stores sensitive metrics about your infrastructure and applications. Without proper security measures, this valuable data could be exposed to unauthorized users, or worse, your monitoring system could become a vulnerability in your infrastructure.

In this guide, we'll explore essential security and authentication practices for Prometheus deployments. You'll learn how to protect your metrics, secure API endpoints, implement authentication, and follow best practices to maintain a robust monitoring system.

Why Prometheus Security Matters​

Before diving into implementation details, let's understand why securing Prometheus is crucial:

  1. Sensitive Data Protection - Prometheus collects metrics that may include sensitive information about your systems
  2. Preventing Unauthorized Access - Without security, anyone with network access could query your metrics or modify your configuration
  3. System Integrity - A compromised monitoring system could lead to undetected issues or attacks
  4. Compliance Requirements - Many regulatory frameworks require secure monitoring practices

Basic Security Concepts in Prometheus​

Prometheus security can be broken down into several key areas:

Securing Prometheus with Network Controls​

Firewall Configuration​

The first line of defense is proper network security. Prometheus servers should not be exposed directly to the internet or untrusted networks.

bash
# Example iptables rules to restrict Prometheus access to specific IPs
iptables -A INPUT -p tcp --dport 9090 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 9090 -j DROP

Using Network Isolation​

Deploy Prometheus in isolated network segments where possible:

yaml
# Example docker-compose.yml with network isolation
version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    networks:
      - monitoring_network
    # ...other configuration...

networks:
  monitoring_network:
    internal: true

Implementing TLS Encryption​

Generating TLS Certificates​

Secure communications with TLS encryption:

bash
# Generate a private key
openssl genrsa -out prometheus.key 2048

# Generate a certificate signing request (CSR)
openssl req -new -key prometheus.key -out prometheus.csr -subj "/CN=prometheus.example.com"

# Generate a self-signed certificate (for testing)
openssl x509 -req -days 365 -in prometheus.csr -signkey prometheus.key -out prometheus.crt

Configuring Prometheus with TLS​

Add TLS configuration to your Prometheus configuration:

yaml
# prometheus.yml
global:
  # ...other global configs...

web:
  tls_server_config:
    cert_file: /etc/prometheus/prometheus.crt
    key_file: /etc/prometheus/prometheus.key

Authentication Methods​

Prometheus itself has limited built-in authentication mechanisms, but there are several approaches to implement authentication:

Using a Reverse Proxy with Basic Auth​

Nginx configuration example:

nginx
server {
    listen 443 ssl;
    server_name prometheus.example.com;

    ssl_certificate /etc/nginx/certs/prometheus.crt;
    ssl_certificate_key /etc/nginx/certs/prometheus.key;

    location / {
        auth_basic "Prometheus";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Generate the password file:

bash
# Create a user 'admin' with password prompt
htpasswd -c /etc/nginx/.htpasswd admin

Using OAuth2 Proxy​

For more advanced authentication, you can use OAuth2 Proxy:

yaml
# docker-compose.yml for OAuth2 Proxy setup
version: '3'
services:
  oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy
    command:
      - --provider=github
      - --email-domain=*
      - --upstream=http://prometheus:9090
      - --http-address=0.0.0.0:4180
      - --cookie-secret=your-secret-here
      - --client-id=your-github-client-id
      - --client-secret=your-github-client-secret
    ports:
      - "4180:4180"
    # ...other configuration...

Prometheus Authorization​

While basic authentication controls who can access your Prometheus instance, authorization controls what they can do once they're in.

Using Alertmanager with Receiver-Based Authorization​

Limit who receives alerts through careful configuration:

yaml
# alertmanager.yml
receivers:
  - name: 'team-db-pager'
    pagerduty_configs:
      - service_key: XXXXXXXX
        routing_key: XXXXXXXX

  - name: 'team-frontend-email'
    email_configs:
      - to: '[email protected]'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'team-db-pager'  # Default receiver
  routes:
  - match:
      service: frontend
    receiver: 'team-frontend-email'

Securing the Prometheus API​

Rate Limiting API Requests​

Protect your Prometheus API endpoints from abuse with rate limiting:

nginx
# Nginx configuration with rate limiting
http {
    limit_req_zone $binary_remote_addr zone=prometheus_api:10m rate=10r/s;
    
    server {
        # ...other server configs...
        
        location /api/ {
            limit_req zone=prometheus_api burst=20 nodelay;
            proxy_pass http://localhost:9090;
        }
    }
}

Implementing API Tokens​

For API access, implement token-based authentication:

bash
# Example using curl with an authentication token
curl -H "Authorization: Bearer your-secure-token-here" https://prometheus.example.com/api/v1/query?query=up

File-Based Security​

Securing Configuration Files​

Protect your Prometheus configuration files with proper permissions:

bash
# Set restrictive permissions on configuration files
chmod 600 prometheus.yml
chmod 600 alertmanager.yml

# Ensure proper ownership
chown prometheus:prometheus prometheus.yml
chown prometheus:prometheus alertmanager.yml

Safe Storage of Secrets​

Never store credentials directly in your Prometheus configuration. Use environment variables or secrets management solutions:

yaml
# Using environment variables in docker-compose.yml
version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    environment:
      - WEBHOOK_URL=${WEBHOOK_SECRET_URL}
    # ...other configuration...

Real-World Security Setup Example​

Let's put everything together with a comprehensive example of a secure Prometheus setup:

yaml
# docker-compose.yml for a secure Prometheus stack
version: '3'

services:
  prometheus:
    image: prom/prometheus:latest
    user: "nobody:nogroup"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./tls:/etc/prometheus/tls:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.console.libraries=/usr/share/prometheus/console_libraries
      - --web.console.templates=/usr/share/prometheus/consoles
      - --web.config.file=/etc/prometheus/web-config.yml
    networks:
      - monitoring

  nginx:
    image: nginx:latest
    ports:
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./tls:/etc/nginx/certs:ro
      - ./.htpasswd:/etc/nginx/.htpasswd:ro
    depends_on:
      - prometheus
    networks:
      - monitoring
      - public

networks:
  monitoring:
    internal: true
  public:

volumes:
  prometheus_data:

With the corresponding web-config.yml for Prometheus:

yaml
# web-config.yml
tls_server_config:
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key

basic_auth_users:
  admin: $2y$10$imBL37R0VyFsZtKqguVouekPJ6dTnUclj8MpCdyKRFvHG0mwdmzd2  # hashed password

Security Best Practices Checklist​

Follow these best practices to ensure a secure Prometheus deployment:

  • ✅ Run Prometheus with minimal privileges
  • ✅ Use TLS encryption for all communications
  • ✅ Implement strong authentication
  • ✅ Regularly update Prometheus and all components
  • ✅ Back up configuration securely
  • ✅ Audit access regularly
  • ✅ Use network isolation
  • ✅ Implement rate limiting for APIs
  • ✅ Follow the principle of least privilege
  • ✅ Monitor the monitoring system itself

Common Security Pitfalls​

Avoid these common security mistakes:

  1. Exposing Prometheus directly to the internet
  2. Using default or weak credentials
  3. Neglecting to encrypt communications
  4. Running Prometheus as root
  5. Storing credentials in plain text configuration files
  6. Not updating Prometheus regularly
  7. Allowing unrestricted access to alerting endpoints

Practical Exercise: Setting Up a Secure Prometheus Instance​

Let's walk through a practical exercise to set up a secure Prometheus instance:

  1. Create a dedicated user for Prometheus:
bash
sudo useradd --no-create-home --shell /bin/false prometheus
  1. Create necessary directories with proper permissions:
bash
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus
  1. Generate TLS certificates and set up configuration:
bash
# Generate certificates as shown earlier
# ...

# Create a secure web-config.yml
cat <<EOF > /etc/prometheus/web-config.yml
tls_server_config:
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key
EOF
sudo chown prometheus:prometheus /etc/prometheus/web-config.yml
sudo chmod 600 /etc/prometheus/web-config.yml
  1. Configure and start Prometheus securely:
bash
# Create basic prometheus.yml
cat <<EOF > /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
EOF
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo chmod 600 /etc/prometheus/prometheus.yml

# Start Prometheus with security options
sudo -u prometheus prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.config.file=/etc/prometheus/web-config.yml

Summary​

Securing your Prometheus installation is an essential aspect of maintaining a robust monitoring system. Through proper implementation of authentication, TLS encryption, network security, and following best practices, you can ensure that your monitoring data remains protected while still being accessible to authorized users.

Remember that security is not a one-time setup but an ongoing process. Regularly review your security configurations, update your systems, and audit access to maintain a secure monitoring environment.

Additional Resources​

Next Steps​

Now that you understand how to secure your Prometheus installation, consider:

  1. Implementing these security measures in your existing Prometheus deployments
  2. Setting up a test environment to practice secure configurations
  3. Creating a security audit checklist specific to your monitoring needs
  4. Exploring advanced topics like mutual TLS authentication

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)