Secret Management

Introduction

When working with Grafana Loki, you'll inevitably handle sensitive information such as API keys, passwords, tokens, and other credentials. These pieces of information are called "secrets," and they require special handling to prevent unauthorized access. Secret management is the practice of securely storing, distributing, rotating, and accessing these sensitive data pieces throughout your infrastructure.

Poor secret management can lead to devastating security breaches, data leaks, and compliance violations. In this guide, we'll explore how to properly manage secrets in Grafana Loki deployments, following industry best practices.

Why Secret Management Matters in Loki

Grafana Loki may interact with various external systems and services, each requiring authentication:

• Storage backends (S3, GCS, Azure Blob Storage)
• Alert managers
• Authentication providers
• Metrics systems
• Other microservices in your stack

Properly managing these secrets is crucial for maintaining a secure logging infrastructure.

Secret Management Principles

Before diving into implementation details, let's understand the core principles of effective secret management:

1. Least Privilege: Secrets should only be accessible to services and individuals who absolutely need them.
2. Encryption at Rest: Secrets should always be encrypted when stored.
3. Encryption in Transit: Secrets should be encrypted when being transferred between systems.
4. Regular Rotation: Secrets should be changed periodically to limit the impact of potential breaches.
5. Audit & Logging: Access to secrets should be tracked and monitored.
6. Automated Management: Reduce human handling of secrets through automation.
7. Centralized Storage: Store secrets in a dedicated, secure system rather than scattered throughout your infrastructure.

Common Secret Management Approaches in Loki

Environment Variables

The simplest approach is using environment variables, though this has significant limitations.

yaml

# Example Docker Compose snippet
version: '3'
services:
  loki:
    image: grafana/loki:latest
    environment:
      - LOKI_S3_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
      - LOKI_S3_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Issues with environment variables:

• They may appear in logs and error messages
• They can be exposed via process listings
• They are difficult to rotate
• They don't scale well across multiple services

Kubernetes Secrets

For Kubernetes deployments, native Secrets objects provide a better alternative.

yaml

# Create a Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: loki-s3-credentials
type: Opaque
data:
  access_key: QUtJQUlPU0ZPRE5ON0VYQU1QTEU=  # base64 encoded
  secret_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQ==  # base64 encoded

And then mount it in your deployment:

yaml

# Mount secret in Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
spec:
  template:
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        env:
        - name: LOKI_S3_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: loki-s3-credentials
              key: access_key
        - name: LOKI_S3_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: loki-s3-credentials
              key: secret_key

Note: By default, Kubernetes Secrets are only base64-encoded, not encrypted. Consider using sealed secrets or an external secret management system for enhanced security.

External Secret Management Solutions

For production environments, dedicated secret management tools are recommended:

1. HashiCorp Vault - A comprehensive secret management solution
2. AWS Secrets Manager/Parameter Store - For AWS deployments
3. Google Secret Manager - For GCP deployments
4. Azure Key Vault - For Azure deployments

Let's look at an example using HashiCorp Vault with Kubernetes:

yaml

# Example using Vault Agent with Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-s3-creds: "secret/data/loki/s3"
        vault.hashicorp.com/role: "loki"
    spec:
      containers:
      - name: loki
        image: grafana/loki:latest
        volumeMounts:
        - name: s3-creds
          mountPath: /etc/secrets/s3
          readOnly: true

Configuring Loki for Secure Secret Management

Using Config Files with Secret References

Loki can be configured to read secrets from files rather than environment variables or direct configuration:

yaml

# loki-config.yaml
storage_config:
  aws:
    s3: s3://bucket-name/loki
    access_key_id_file: /etc/secrets/s3/access-key
    secret_access_key_file: /etc/secrets/s3/secret-key

This approach allows you to:

1. Mount secrets as files from your secret management system
2. Rotate secrets without restarting Loki (if implemented properly)
3. Keep secrets out of your configuration files

Implementing Secret Rotation

Secret rotation is a crucial security practice. Here's a basic approach for Loki:

1. Create new credentials in your external service (e.g., AWS)
2. Update your secret management system with the new credentials
3. Configure your deployment to automatically pick up the new credentials
4. Verify the new credentials work
5. Revoke the old credentials

For Kubernetes deployments with Vault, this can be automated:

yaml

# Example Vault policy that rotates AWS keys
path "aws/creds/loki-role" {
  capabilities = ["read"]
}

Best Practices for Loki Secret Management

1. Never commit secrets to version control

   bash

   # Use .gitignore to prevent accidental commits
   echo "*.key" >> .gitignore
   echo "*.pem" >> .gitignore
   echo "*.env" >> .gitignore

2. Use CI/CD pipelines to inject secrets during deployment

   yaml

   # Example GitHub Actions workflow (partial)
   steps:
     - name: Configure AWS credentials
       uses: aws-actions/configure-aws-credentials@v1
       with:
         aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
         aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

3. Implement secret rotation policies

   Define and enforce regular rotation schedules:

   • API keys: Every 90 days
   • Database credentials: Every 30-90 days
   • TLS certificates: Before expiration

4. Audit secret access

   Regularly review who and what has access to your secrets:

   bash

   # Example Vault audit command
   vault audit list

5. Use different secrets for different environments

   Never share secrets between development, staging, and production.

Troubleshooting Secret Management Issues

Common issues and solutions:

Issue	Possible Cause	Solution
"Access Denied" errors	Expired/rotated credentials	Check if credentials have been rotated and update accordingly
Permission errors	Insufficient IAM/RBAC permissions	Review and adjust permissions using principle of least privilege
Startup failures	Secret not available/accessible	Verify secret exists and is mounted correctly
Performance degradation	Excessive external secret fetching	Implement caching or sidecars for secret retrieval

Real-World Example: Complete Loki Deployment with Secrets

Here's a comprehensive example using Kubernetes with external secrets:

yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: loki-sa
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki
spec:
  serviceName: loki
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-s3-creds: "secret/data/loki/s3"
        vault.hashicorp.com/agent-inject-template-s3-creds: |
          {{- with secret "secret/data/loki/s3" -}}
          [default]
          aws_access_key_id = {{ .Data.data.access_key }}
          aws_secret_access_key = {{ .Data.data.secret_key }}
          {{- end }}
    spec:
      serviceAccountName: loki-sa
      containers:
      - name: loki
        image: grafana/loki:latest
        ports:
        - containerPort: 3100
          name: http
        volumeMounts:
        - name: config
          mountPath: /etc/loki
        - name: data
          mountPath: /data
        env:
        - name: AWS_SHARED_CREDENTIALS_FILE
          value: /vault/secrets/s3-creds
      volumes:
      - name: config
        configMap:
          name: loki-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Summary

Effective secret management is essential for securing your Grafana Loki deployment. Key takeaways:

• Always store secrets securely using dedicated secret management solutions
• Follow the principle of least privilege
• Implement regular secret rotation
• Never hardcode secrets in your application code or configuration files
• Use different secrets for different environments
• Audit and monitor secret access

By implementing these practices, you'll significantly reduce the risk of security breaches and unauthorized access to your logging infrastructure.

Additional Resources

• HashiCorp Vault Documentation
• Kubernetes Secrets Documentation
• OWASP Secret Management Cheat Sheet

Exercises

1. Set up a local Loki deployment using Docker Compose with environment variables for secrets. Then, refactor it to use Docker secrets instead.

2. If you have a Kubernetes cluster, deploy Loki using Kubernetes Secrets, and then enhance it by integrating with an external secret management tool like HashiCorp Vault.

3. Create a secret rotation plan for your Loki deployment, detailing which secrets need rotation, frequency, and process.

4. Audit your current Loki configuration for hardcoded secrets and create an action plan to eliminate them.