Configuration Validation

Introduction

One of the most common sources of problems with Prometheus is misconfiguration. Even small syntax errors or logical mistakes in your configuration files can prevent Prometheus from starting up properly or cause it to behave unexpectedly. In this guide, we'll explore how to validate your Prometheus configurations, identify common errors, and use built-in tools to ensure your monitoring setup is correct.

Configuration validation is an essential skill for anyone working with Prometheus. It helps you:

Identify syntax errors before they cause downtime
Verify that your scrape configurations will work as expected
Ensure your alerting rules are properly formatted
Catch logical errors in your recording rules

Let's dive into how you can validate your Prometheus configurations effectively.

Basic Configuration Validation

Prometheus provides a built-in command to check the syntax of your configuration files without actually starting the service. This is incredibly useful for catching basic errors before deploying your changes.

Using `promtool`

promtool is a utility that comes bundled with Prometheus and provides several helpful functions, including configuration validation.

To check your main Prometheus configuration file:

bash
promtool check config prometheus.yml

If your configuration is valid, you'll see output similar to:

checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file

If there are errors, promtool will provide specific information about what's wrong:

checking prometheus.yml
  FAILED: error parsing prometheus.yml: yaml: line 42: did not find expected key

Validating Rules

Similarly, you can validate your alerting and recording rules:

bash
promtool check rules rules.yml

A successful validation will show:

Checking rules.yml
  SUCCESS: 2 rules found

While an error might look like:

Checking rules.yml
  FAILED: error parsing rules.yml: yaml: line 15: did not find expected key

Common Configuration Errors

Let's explore some of the most common configuration errors and how to identify and fix them.

YAML Syntax Errors

YAML is sensitive to indentation and formatting. Common YAML errors include:

Incorrect indentation
Missing colons after keys
Using tabs instead of spaces
Unquoted strings containing special characters

Example of incorrect YAML:

yaml
scrape_configs:
  - job_name: node
    static_configs
      - targets: ['localhost:9100']

Corrected version:

yaml
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Notice the missing colon after static_configs in the incorrect version.

Invalid Relabeling Configurations

Relabeling is a powerful feature but is also a common source of errors.

Example of incorrect relabeling:

yaml
relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):(.*)'
    replacement: '${1}'
    target_label: instance
    action: unknown_action

Corrected version:

yaml
relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):(.*)'
    replacement: '${1}'
    target_label: instance
    action: replace

The action value must be one of the supported actions (replace, keep, drop, etc.).

Prometheus Rule Syntax Errors

Alert and recording rules must follow specific syntax requirements.

Example of incorrect rule:

yaml
groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: job:request_errors:rate5m / job:requests:rate5m > 0.1
        for: 10m
        lables:
          severity: warning

Corrected version:

yaml
groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: job:request_errors:rate5m / job:requests:rate5m > 0.1
        for: 10m
        labels:
          severity: warning

Notice the typo in lables (should be labels).

Advanced Configuration Validation

Beyond basic syntax checking, there are more sophisticated ways to validate your configuration.

Dry Run Mode

You can start Prometheus in a "dry run" mode where it loads the configuration but exits immediately without starting all components:

bash
prometheus --config.file=prometheus.yml --enable-feature=promql-at-modifier,promql-negative-offset --web.listen-address=0.0.0.0:9090

Watch the output for any warnings or errors. This will catch more subtle issues that the basic config check might miss.

Validation of PromQL Expressions

Prometheus doesn't automatically validate the correctness of your PromQL expressions in rules. You can manually test these against your running Prometheus server:

bash
promtool query instant http://localhost:9090 'job:request_errors:rate5m / job:requests:rate5m > 0.1'

This helps ensure that your alert and recording rule expressions are valid.

Visual Configuration Validation

Let's visualize a typical configuration validation workflow: