Authentication Options

Introduction

Authentication is a critical component of any security strategy for your Grafana Loki deployment. It verifies the identity of users or services attempting to access your Loki instance, ensuring that only authorized entities can interact with your log data. This guide explores the various authentication options available in Grafana Loki, helping you choose and implement the most appropriate authentication method for your environment.

Authentication in Loki serves several important purposes:

• Preventing unauthorized access to sensitive log data
• Enabling multi-tenant deployments where different teams access only their own logs
• Tracking and auditing who has accessed what data
• Integrating with your organization's existing identity management systems

Basic Authentication Concepts

Before diving into Loki-specific authentication options, let's understand some fundamental authentication concepts:

Authentication vs. Authorization

While often used interchangeably, these terms refer to different aspects of security:

• Authentication: Verifies who you are (identity)
• Authorization: Determines what you're allowed to do (permissions)

Loki implements both, but this guide focuses primarily on authentication methods.

Multi-Factor Authentication (MFA)

Multi-factor authentication combines multiple verification methods:

• Something you know (password)
• Something you have (token device)
• Something you are (biometrics)

While Loki itself doesn't directly implement MFA, it can integrate with identity providers that do.

Authentication Methods in Grafana Loki

Loki supports several authentication mechanisms depending on your deployment model and security requirements.

No Authentication (Insecure)

For development or isolated environments, Loki can be configured without authentication:

yaml

auth_enabled: false

caution

Running Loki without authentication is only recommended for development environments. Never deploy Loki without authentication in production environments.

Basic Authentication

Basic authentication uses a username and password combination sent in HTTP headers. In Loki, this is typically configured through a proxy or gateway service.

Example of configuring basic auth with Nginx as a reverse proxy:

nginx

server {
    listen 80;
    server_name loki.example.com;

    location / {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:3100;
    }
}

To create the password file:

bash

htpasswd -c /etc/nginx/.htpasswd admin

Token-Based Authentication

Loki supports token-based authentication, where clients include an API token in their requests.

Configuration example in Loki's configuration file:

yaml

auth_enabled: true

server:
  http_listen_port: 3100

distributor:
  ring:
    kvstore:
      store: inmemory

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

# Authentication configuration
auth:
  type: token
  token:
    enabled: true
    header: X-Scope-OrgID

With this configuration, clients must include the X-Scope-OrgID header with a valid token:

bash

curl -H "X-Scope-OrgID: tenant1" -X GET http://localhost:3100/loki/api/v1/labels

OAuth 2.0 Integration

For enterprise environments, OAuth 2.0 is often preferred. This allows Loki to delegate authentication to an identity provider like Google, GitHub, or an internal OAuth provider.

Typically, this is set up through a proxy that handles the OAuth flow, such as OAuth2 Proxy:

yaml

# Example docker-compose.yml for Loki with OAuth2 Proxy
version: '3'
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml

  oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    ports:
      - "4180:4180"
    environment:
      - OAUTH2_PROXY_CLIENT_ID=your_client_id
      - OAUTH2_PROXY_CLIENT_SECRET=your_client_secret
      - OAUTH2_PROXY_COOKIE_SECRET=random_cookie_secret
      - OAUTH2_PROXY_EMAIL_DOMAINS=*
      - OAUTH2_PROXY_PROVIDER=github
      - OAUTH2_PROXY_UPSTREAMS=http://loki:3100/
      - OAUTH2_PROXY_HTTP_ADDRESS=0.0.0.0:4180

Multi-Tenancy with Authentication

Loki supports multi-tenancy, where different teams or applications can use the same Loki instance but only access their own data.

Configuration example for multi-tenancy:

yaml

auth_enabled: true

# Other config sections...

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_global_streams_per_user: 5000
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

# Multi-tenancy configuration with per-tenant limits
multi_tenancy_enabled: true
tenant_configs:
  configs:
    - tenant: team1
      limits:
        ingestion_rate_mb: 5
        ingestion_burst_size_mb: 10
    - tenant: team2
      limits:
        ingestion_rate_mb: 8
        ingestion_burst_size_mb: 16

Implementing JWT Authentication

JSON Web Tokens (JWT) provide a secure way to transmit authentication information between parties. Here's how to set up JWT authentication with Loki:

1. Configure a proxy to validate JWTs and inject the tenant ID into a header

javascript

// Example Express.js middleware for JWT validation
const jwt = require('jsonwebtoken');
const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const app = express();
const SECRET_KEY = process.env.JWT_SECRET;

// JWT validation middleware
app.use((req, res, next) => {
  const authHeader = req.headers.authorization;
  
  if (!authHeader || !authHeader.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'No token provided' });
  }
  
  const token = authHeader.split(' ')[1];
  
  try {
    const decoded = jwt.verify(token, SECRET_KEY);
    // Add tenant ID from JWT claim
    req.headers['X-Scope-OrgID'] = decoded.tenant;
    next();
  } catch (err) {
    return res.status(401).json({ error: 'Invalid token' });
  }
});

// Proxy requests to Loki
app.use('/', createProxyMiddleware({
  target: 'http://loki:3100',
  changeOrigin: true
}));

app.listen(8080, () => {
  console.log('JWT Proxy listening on port 8080');
});

2. Configure clients to include the JWT in the Authorization header

bash

# Example curl request with JWT
JWT="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZW5hbnQiOiJ0ZWFtMSIsInN1YiI6InVzZXIxIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"

curl -H "Authorization: Bearer $JWT" \
  http://localhost:8080/loki/api/v1/query_range \
  --data-urlencode "query={app=\"myapp\"}" \
  --data-urlencode "start=1609455600000000000" \
  --data-urlencode "end=1609459200000000000"

Authentication with Grafana as a Frontend

Most users interact with Loki via Grafana, which handles authentication on behalf of users.

To configure Loki as a data source in Grafana with authentication:

1. In Grafana, go to Configuration > Data Sources > Add data source
2. Select Loki
3. Enter the URL of your Loki instance
4. Under Auth, configure the authentication method
5. For multi-tenancy, add Custom Headers:
   • Name: X-Scope-OrgID
   • Value: The tenant ID

Best Practices for Authentication

1. Never run Loki without authentication in production
2. Use HTTPS/TLS for all communications to prevent credential interception
3. Implement proper token rotation for API keys and tokens
4. Use the principle of least privilege when granting access
5. Regularly audit authentication logs to detect suspicious activity
6. Integrate with your organization's existing identity management where possible
7. Use environment variables for sensitive authentication information

Troubleshooting Authentication Issues

Common Problems and Solutions

Problem	Possible Causes	Solution
"Unauthorized" errors	Missing or invalid token	Check that the X-Scope-OrgID header is being sent
"Forbidden" errors	Valid authentication but insufficient permissions	Check tenant configuration and permissions
Authentication working inconsistently	Load balancer or proxy issues	Ensure session affinity if needed

Debugging Authentication

To troubleshoot authentication issues, enable debug logging in Loki:

yaml

server:
  http_listen_port: 3100
  log_level: debug

Then check logs for authentication-related messages:

bash

docker logs loki-container 2>&1 | grep -i auth

Summary

Authentication is a critical aspect of securing your Grafana Loki deployment. This guide has covered:

• Basic authentication concepts
• Various authentication methods supported by Loki
• Implementation examples for different authentication options
• Multi-tenancy considerations
• JWT authentication setup
• Integration with Grafana
• Best practices and troubleshooting

By understanding and properly implementing these authentication options, you can ensure that your log data remains secure while being accessible to authorized users and services.

Additional Resources

• Grafana Loki GitHub Repository
• Grafana Loki Documentation
• OAuth2 Proxy Project

Exercises

1. Set up Loki with basic authentication using an Nginx reverse proxy
2. Implement token-based authentication and test access with different tenant IDs
3. Configure Grafana to connect to Loki with the appropriate authentication headers
4. Create a simple JWT proxy using the example code and test authentication
5. Design a multi-tenant authentication strategy for a hypothetical organization with three teams