HAProxy Exporter

Introduction

HAProxy (High Availability Proxy) is a popular open-source load balancer and proxy server for TCP and HTTP-based applications. When running applications in production, monitoring the health and performance of your load balancers is crucial for ensuring system reliability.

The HAProxy Exporter is a Prometheus exporter that collects metrics from HAProxy instances and exposes them in a format that Prometheus can scrape. This allows you to monitor key performance indicators of your HAProxy instances, set up alerts for potential issues, and visualize the data using tools like Grafana.

In this guide, we'll explore:

• What HAProxy Exporter is and why it's useful
• How to install and configure HAProxy Exporter
• Understanding the metrics it provides
• Setting up Prometheus to scrape these metrics
• Real-world examples of monitoring HAProxy

Prerequisites

Before we begin, you should have:

• A basic understanding of Prometheus
• HAProxy installed and running
• Prometheus server installed and operational
• Basic knowledge of Linux commands

What is HAProxy Exporter?

HAProxy Exporter is an official Prometheus exporter that converts HAProxy statistics into Prometheus metrics. HAProxy provides a statistics page that shows current proxy status, but the HAProxy Exporter transforms this information into time-series data that can be stored, queried, and visualized.

Here's how it fits into your monitoring architecture:

Installation and Setup

Installing HAProxy Exporter

You can install HAProxy Exporter in several ways:

Using Binary Release

bash

# Download the latest release (check for the most current version)
wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.14.0/haproxy_exporter-0.14.0.linux-amd64.tar.gz

# Extract the binary
tar xvzf haproxy_exporter-0.14.0.linux-amd64.tar.gz

# Move the binary to a directory in your PATH
sudo mv haproxy_exporter-0.14.0.linux-amd64/haproxy_exporter /usr/local/bin/

# Make it executable
sudo chmod +x /usr/local/bin/haproxy_exporter

Using Docker

bash

docker run -d --name haproxy-exporter \
  -p 9101:9101 \
  prom/haproxy-exporter:latest \
  --haproxy.scrape-uri="http://user:password@haproxy-host:port/haproxy?stats;csv"

Configuring HAProxy for Metrics Collection

Before the exporter can collect metrics, you need to enable the statistics page in HAProxy. Add the following to your HAProxy configuration file (/etc/haproxy/haproxy.cfg):

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

After making these changes, restart HAProxy:

bash

sudo systemctl restart haproxy

Running HAProxy Exporter

To start the HAProxy Exporter, run:

bash

haproxy_exporter --haproxy.scrape-uri="http://localhost:8404/stats;csv"

This command tells the exporter to fetch metrics from the HAProxy stats page at the specified URL. The exporter will start listening on port 9101 by default.

For production environments, it's recommended to set up HAProxy Exporter as a system service:

bash

# Create a system user for the exporter
sudo useradd --no-create-home --shell /bin/false haproxy_exporter

# Create a systemd service file
sudo nano /etc/systemd/system/haproxy_exporter.service

Add the following content to the service file:

ini

[Unit]
Description=HAProxy Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=haproxy_exporter
Group=haproxy_exporter
Type=simple
ExecStart=/usr/local/bin/haproxy_exporter --haproxy.scrape-uri="http://localhost:8404/stats;csv"

[Install]
WantedBy=multi-user.target

Then enable and start the service:

bash

sudo systemctl daemon-reload
sudo systemctl enable haproxy_exporter
sudo systemctl start haproxy_exporter

Configuring Prometheus to Scrape HAProxy Metrics

Now that the HAProxy Exporter is running, you need to configure Prometheus to scrape the metrics. Add the following job to your Prometheus configuration file (prometheus.yml):

yaml

scrape_configs:
  - job_name: 'haproxy'
    static_configs:
      - targets: ['localhost:9101']

After saving the configuration, restart Prometheus:

bash

sudo systemctl restart prometheus

Understanding HAProxy Metrics

HAProxy Exporter provides numerous metrics. Here are some of the most important ones:

Frontend Metrics

• haproxy_frontend_current_sessions: Number of current sessions
• haproxy_frontend_bytes_in_total: Total number of bytes received
• haproxy_frontend_bytes_out_total: Total number of bytes sent
• haproxy_frontend_request_errors_total: Total number of request errors
• haproxy_frontend_http_responses_total: Total number of HTTP responses, partitioned by status code

Backend Metrics

• haproxy_backend_current_sessions: Number of current sessions
• haproxy_backend_bytes_in_total: Total number of bytes received
• haproxy_backend_bytes_out_total: Total number of bytes sent
• haproxy_backend_connection_errors_total: Total number of connection errors
• haproxy_backend_response_errors_total: Total number of response errors
• haproxy_backend_up: Current health status of the backend (1 = UP, 0 = DOWN)

Server Metrics

• haproxy_server_current_sessions: Number of current sessions
• haproxy_server_bytes_in_total: Total number of bytes received
• haproxy_server_bytes_out_total: Total number of bytes sent
• haproxy_server_connection_errors_total: Total number of connection errors
• haproxy_server_up: Current health status of the server (1 = UP, 0 = DOWN)

Practical Examples

Example 1: Monitoring Frontend Connection Rate

The following PromQL query shows the rate of new connections per second to your frontends:

rate(haproxy_frontend_connections_total[5m])

This is useful for understanding traffic patterns and capacity planning.

Example 2: Detecting Backend Failures

This query shows whether each backend is up (1) or down (0):

haproxy_backend_up

You can use this to detect backend failures and set up alerts.

Example 3: HTTP Error Rate

The following query calculates the ratio of HTTP 5xx errors to total requests:

sum by (frontend) (rate(haproxy_frontend_http_responses_total{code="5xx"}[5m])) / 
sum by (frontend) (rate(haproxy_frontend_http_responses_total[5m]))

This helps monitor the health of your application by tracking error rates.

Example 4: Setting Up an Alert for Backend Failures

You can add the following alert rule to your Prometheus configuration:

yaml

groups:
- name: haproxy
  rules:
  - alert: HAProxyBackendDown
    expr: haproxy_backend_up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "HAProxy backend down"
      description: "Backend {{ $labels.backend }} has been down for more than 1 minute."

This will alert you when any backend is down for more than a minute.

Visualizing HAProxy Metrics with Grafana

Once you have Prometheus scraping HAProxy metrics, you can create dashboards in Grafana to visualize them. Here's a simple example of a Grafana dashboard query to show current sessions across all frontends:

1. Add your Prometheus data source in Grafana
2. Create a new dashboard
3. Add a new panel
4. Use the following query:

sum by (frontend) (haproxy_frontend_current_sessions)

For a more comprehensive monitoring setup, you can import existing HAProxy dashboards from the Grafana dashboard marketplace.

Troubleshooting HAProxy Exporter

Common Issues

1. Cannot connect to HAProxy stats page

   • Verify HAProxy is running: sudo systemctl status haproxy
   • Check that the stats page is configured correctly
   • Ensure the URL in the exporter configuration is correct

2. No metrics showing in Prometheus

   • Verify the exporter is running: sudo systemctl status haproxy_exporter
   • Check that Prometheus is correctly configured to scrape the exporter
   • Look for errors in the Prometheus logs: sudo journalctl -u prometheus

3. Authorization issues with stats page

   • If your stats page requires authentication, ensure the credentials are correctly set in the scrape URL

Best Practices

1. Security: Restrict access to the HAProxy stats page using authentication and firewall rules
2. Performance: Monitor the resource usage of HAProxy Exporter itself
3. Alerting: Set up alerts for critical metrics to catch issues before they affect users
4. Dashboard Organization: Create separate dashboard sections for frontends, backends, and servers

Summary

HAProxy Exporter is a powerful tool for monitoring your HAProxy instances. By collecting metrics from HAProxy and exposing them in a format compatible with Prometheus, it enables you to:

• Track the performance and health of your load balancers
• Detect and respond to issues quickly
• Understand traffic patterns and resource utilization
• Make data-driven decisions about scaling and optimization

With the configuration steps outlined in this guide, you should now have a solid foundation for monitoring your HAProxy instances with Prometheus.

Additional Resources

• Official HAProxy Exporter GitHub
• HAProxy Documentation
• Prometheus Documentation
• Grafana Dashboard for HAProxy

Exercises

1. Install HAProxy Exporter and configure it to collect metrics from a local HAProxy instance.
2. Create a Prometheus alert that notifies you when any backend server is down for more than 5 minutes.
3. Design a Grafana dashboard that shows the key performance metrics for your HAProxy instance.
4. Use PromQL to find the frontend with the highest rate of 5xx errors over the last hour.
5. Configure HAProxy Exporter to collect metrics from multiple HAProxy instances and label them appropriately.