Data Source Security

Introduction

Data sources are the backbone of any Grafana deployment, connecting your dashboards to various metrics, logs, and other information systems. However, these connections can also represent significant security vulnerabilities if not properly configured and maintained. In this guide, we'll explore the fundamentals of securing your Grafana data sources to protect both your monitoring infrastructure and the valuable data it accesses.

Why Data Source Security Matters

When you connect Grafana to databases, cloud services, or other systems, you're creating pathways that could potentially be exploited. Consider these risks:

Credentials stored in data source configurations could be compromised
Insecure connections might expose sensitive data in transit
Overly permissive data source permissions could lead to unauthorized data access
Insecure API keys or tokens might grant excessive system access

Let's explore how to mitigate these risks through proper configuration and security practices.

Secure Authentication for Data Sources

Using Encrypted Credentials

Grafana encrypts data source credentials using a database encryption key. This key is generated automatically when Grafana starts for the first time and stored in the Grafana database.

bash
# Example setting in grafana.ini to customize database encryption key
[security]
# previous default (8 characters)
secret_key = SW2YcwTIb9zpOOhoPsMm
# recommended length (32 characters minimum)
secret_key = AAABBBCCCDDDEEEFFFGGGHHHIIIJJJKKKLLLMMMMNNNOOOPPPQQQRRRSSSSTTTUUUVVVWWWXXXYYYZZZabcdefghijklm

Using Environment Variables for Secrets

Rather than hardcoding credentials in Grafana's database, use environment variables:

bash
# Setting database password as environment variable
GF_DATASOURCES_DATABASE_PASSWORD=secure_password_here

In your data source configuration:

json
{
  "name": "My SQL Server",
  "type": "mssql",
  "url": "localhost:1433",
  "user": "grafana",
  "password": "${GF_DATASOURCES_DATABASE_PASSWORD}",
  "database": "metrics"
}

Implementing Transport Layer Security

Always use encrypted connections to your data sources whenever possible.

Configuring TLS/SSL for Database Connections

Most data sources support TLS/SSL connections. Here's an example with PostgreSQL:

json
{
  "name": "Secure Postgres",
  "type": "postgres",
  "url": "postgres-server:5432",
  "user": "grafana_user",
  "secureJsonData": {
    "password": "secure_password_here"
  },
  "jsonData": {
    "sslmode": "verify-full",
    "tlsAuth": true,
    "tlsAuthWithCACert": true
  },
  "secureJsonData": {
    "tlsCACert": "-----BEGIN CERTIFICATE-----
MIIFADCC...
-----END CERTIFICATE-----",
    "tlsClientCert": "-----BEGIN CERTIFICATE-----
MIIFBDCC...
-----END CERTIFICATE-----",
    "tlsClientKey": "-----BEGIN RSA PRIVATE KEY-----
MIIEpQ...
-----END RSA PRIVATE KEY-----"
  }
}

Handling Self-Signed Certificates

For development environments or internal systems with self-signed certificates:

json
{
  "name": "Internal Prometheus",
  "type": "prometheus",
  "url": "https://internal-prometheus:9090",
  "jsonData": {
    "tlsSkipVerify": true
  }
}

⚠️ Warning: Skipping TLS verification should be avoided in production environments as it makes your connection vulnerable to man-in-the-middle attacks.

Data Source Permissions

Grafana allows granular control over which users can access which data sources through its permissions system.

Implementing Role-Based Access Control (RBAC)

Since Grafana v8.0, RBAC enables you to restrict data source access:

Configuring Data Source Permissions

Navigate to Configuration → Data sources
Select the data source you want to configure
Click the Permissions tab
Add permissions for users, teams, or roles

Example permission setup through provisioning:

yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    version: 1
    # Define permissions
    permissions:
      - role: "Viewer"
        permission: "Query"
      - role: "Editor"
        permission: "Query"
      - role: "Editor"
        permission: "Admin"
      - teamId: 1
        permission: "Admin"

Principle of Least Privilege

When configuring data source credentials, follow the principle of least privilege.

Creating Limited Database Users

Example of creating a read-only MySQL user for Grafana:

sql
-- Create a read-only MySQL user for Grafana
CREATE USER 'grafana_reader'@'%' IDENTIFIED BY 'strong_password_here';
GRANT SELECT ON metrics_database.* TO 'grafana_reader'@'%';
FLUSH PRIVILEGES;

Using Service Accounts for Cloud Services

For cloud data sources, create dedicated service accounts with minimal permissions:

bash
# AWS CLI example creating a read-only IAM user for CloudWatch
aws iam create-user --user-name grafana-cloudwatch-reader
aws iam attach-user-policy --user-name grafana-cloudwatch-reader --policy-arn arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess

Secure API Key Management

Many data sources require API keys for authentication.

API Key Rotation

Regularly rotate API keys used for data sources:

Using Vault for Secret Management

For enterprise environments, integrate with HashiCorp Vault or similar secret management systems:

bash
# Example using Vault for database credentials
vault write database/config/my-postgresql-database \
    plugin_name=postgresql-database-plugin \
    allowed_roles="grafana-role" \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/postgres?sslmode=disable" \
    username="admin" \
    password="adminpassword"

# Create a role with limited permissions
vault write database/roles/grafana-role \
    db_name=my-postgresql-database \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Monitoring Data Source Access

Set up alerts for unusual data source activity.

Auditing Data Source Usage

Enable Grafana's audit logs to track data source access:

ini
# grafana.ini configuration
[auditing]
enabled = true
log_outputs = file
# ... other audit settings

Example audit log entry for data source access:

t=2023-04-18T10:15:30+0000 level=info logger=auditing.auditor action=datasources.query user=admin org=1 datasource=Prometheus

Setting Up Alerts for Unusual Activity

Create alerts based on the audit logs using Loki or other log monitoring tools:

sum(count_over_time({filename="/var/log/grafana/grafana-audit.log"} |= "action=datasources.query" |= "user=admin" [5m])) > 100

Security Scanning and Vulnerability Management

Regularly check for vulnerabilities in your data sources and Grafana installation.

Using Open Source Security Scanners

Tools like Trivy can scan your Grafana container for vulnerabilities:

bash
# Example Trivy scan command
trivy image grafana/grafana:latest

Keeping Data Sources Updated

Maintain a regular update schedule for both Grafana and your data sources:

bash
# Example updating a Prometheus container
docker pull prom/prometheus:latest
docker stop prometheus
docker rm prometheus
docker run -d --name prometheus -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:latest

Network Security for Data Sources

Implement network-level protections for your data sources.

Using Network Isolation

Place data sources in private networks whenever possible:

Implementing IP Whitelisting

Configure firewalls to only allow connections from Grafana servers:

bash
# Example iptables rules to allow only Grafana server access
iptables -A INPUT -p tcp -s grafana_server_ip --dport 5432 -j ACCEPT
iptables -A INPUT -p tcp --dport 5432 -j DROP

Practical Example: Securing a Complete Grafana Stack

Let's walk through a comprehensive example of securing a Grafana stack with multiple data sources.

Architecture Overview

Sample Docker Compose Configuration

yaml
version: '3'
services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/admin_password
      - GF_DATABASE_TYPE=postgres
      - GF_DATABASE_HOST=postgres:5432
      - GF_DATABASE_NAME=grafana
      - GF_DATABASE_USER=grafana
      - GF_DATABASE_PASSWORD__FILE=/run/secrets/db_password
      - GF_DATABASE_SSL_MODE=verify-full
      - GF_DATABASE_CA_CERT_PATH=/etc/grafana/certs/ca.pem
      - GF_SECURITY_ENCRYPTION_PROVIDER=aes-gcm
      - GF_SECURITY_ENCRYPTION_SECRET_KEY__FILE=/run/secrets/encryption_key
    volumes:
      - ./certs:/etc/grafana/certs:ro
      - ./provisioning:/etc/grafana/provisioning:ro
    secrets:
      - admin_password
      - db_password
      - encryption_key
    networks:
      - frontend
      - backend

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./certs:/etc/prometheus/certs:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.config.file=/etc/prometheus/certs/web-config.yml'
    networks:
      - backend

  postgres:
    image: postgres:latest
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./certs:/var/lib/pki:ro
    command: >
      -c ssl=on
      -c ssl_cert_file=/var/lib/pki/server.pem
      -c ssl_key_file=/var/lib/pki/server.key
      -c ssl_ca_file=/var/lib/pki/ca.pem
    secrets:
      - postgres_password
    networks:
      - backend

networks:
  frontend:
  backend:
    internal: true

volumes:
  postgres-data:

secrets:
  admin_password:
    file: ./secrets/admin_password.txt
  db_password:
    file: ./secrets/db_password.txt
  postgres_password:
    file: ./secrets/postgres_password.txt
  encryption_key:
    file: ./secrets/encryption_key.txt

Provisioning Secure Data Sources

yaml
# datasources.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: https://prometheus:9090
    jsonData:
      httpMethod: GET
      tlsAuth: true
      tlsAuthWithCACert: true
      tlsSkipVerify: false
    secureJsonData:
      tlsCACert: ${__file {/etc/grafana/certs/ca.pem}}
      tlsClientCert: ${__file {/etc/grafana/certs/client.pem}}
      tlsClientKey: ${__file {/etc/grafana/certs/client.key}}
    editable: false
    permissions:
      - role: "Viewer"
        permission: "Query"
      - role: "Admin"
        permission: "Admin"

  - name: PostgreSQL
    type: postgres
    url: postgres:5432
    database: metrics
    user: grafana_reader
    secureJsonData:
      password: ${__file {/run/secrets/metrics_db_password}}
    jsonData:
      sslmode: "verify-full"
      tlsAuthWithCACert: true
    secureJsonData:
      tlsCACert: ${__file {/etc/grafana/certs/ca.pem}}
    editable: false

Summary

Securing your Grafana data sources is a critical aspect of maintaining a robust monitoring infrastructure. By implementing proper authentication, encryption, access controls, and network security measures, you can significantly reduce the risk of unauthorized access to your monitoring data and the systems it connects to.

Remember these key principles:

Always encrypt credentials and use environment variables or secret management systems
Implement TLS/SSL for all data source connections
Apply the principle of least privilege to data source accounts
Regularly rotate API keys and credentials
Set up audit logging and monitoring for data source access
Keep all components updated and scan for vulnerabilities
Isolate data sources in private networks when possible

By following these best practices, you can build a secure Grafana deployment that protects both your monitoring infrastructure and the valuable data it accesses.

Additional Resources

Practice Exercises

Set up a Grafana test environment with at least two data sources, implementing all the security measures described in this guide.
Create a rotation schedule for all credentials used in your Grafana deployment.
Implement an audit logging system that alerts on suspicious data source access patterns.
Conduct a security assessment of your current Grafana deployment and identify any vulnerabilities.
Document a security incident response plan specific to your Grafana monitoring infrastructure.

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction​

Why Data Source Security Matters​

Secure Authentication for Data Sources​

Using Encrypted Credentials​

Using Environment Variables for Secrets​

Implementing Transport Layer Security​

Configuring TLS/SSL for Database Connections​

Handling Self-Signed Certificates​

Data Source Permissions​

Implementing Role-Based Access Control (RBAC)​

Configuring Data Source Permissions​

Principle of Least Privilege​

Creating Limited Database Users​

Using Service Accounts for Cloud Services​

Secure API Key Management​

API Key Rotation​

Using Vault for Secret Management​

Monitoring Data Source Access​

Auditing Data Source Usage​

Setting Up Alerts for Unusual Activity​

Security Scanning and Vulnerability Management​

Using Open Source Security Scanners​

Keeping Data Sources Updated​

Network Security for Data Sources​

Using Network Isolation​

Implementing IP Whitelisting​

Practical Example: Securing a Complete Grafana Stack​

Architecture Overview​

Sample Docker Compose Configuration​

Provisioning Secure Data Sources​

Summary​

Additional Resources​

Practice Exercises​