Object Storage (S3/GCS)

Introduction

Object storage is a critical component of any scalable logging infrastructure. When working with Grafana Loki, configuring proper storage backends ensures your logs are stored efficiently, durably, and cost-effectively. This guide focuses on using cloud-based object storage solutions like Amazon S3 (Simple Storage Service) and Google Cloud Storage (GCS) with Loki.

Object storage differs from traditional file systems or block storage by treating data as discrete units (objects) that include the data itself, metadata, and a unique identifier. This approach makes it ideal for storing large volumes of log data that doesn't change once written.

Why Use Object Storage with Loki?

Loki's architecture separates the write path from the read path, allowing it to efficiently store logs in object storage while maintaining fast query capabilities. Here are key benefits:

• Scalability: Easily scale to store petabytes of log data
• Cost-efficiency: Pay only for what you use with no upfront hardware costs
• Durability: Cloud providers typically offer 99.999999999% (11 nines) durability
• Separation of concerns: Isolate storage from compute resources
• Disaster recovery: Geographical replication options

Storage Components in Loki

Before we dive into specific object storage configurations, let's understand how Loki organizes data:

Loki stores two types of data:

1. Chunks: The actual compressed log data
2. Indexes: Metadata that helps locate chunks quickly

Both can be stored in object storage, though indexes might also be stored in databases like Cassandra or BoltDB for performance reasons.

Amazon S3 Configuration

Amazon S3 is one of the most popular object storage solutions and works seamlessly with Grafana Loki.

Basic S3 Setup

To configure Loki with S3, you'll need to add the following to your Loki configuration file:

yaml

storage_config:
  aws:
    s3: s3://access_key:secret_key@region/bucket_name
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

Detailed Configuration

For production environments, you'll want more control over your S3 configuration:

yaml

storage_config:
  aws:
    bucketnames: loki-data
    region: us-west-2
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    s3forcepathstyle: true
    insecure: false
    sse_encryption: true
    http:
      idle_conn_timeout: 90s
      response_header_timeout: 0s
      insecure_skip_verify: false
    s3: null
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
    shared_store: s3

IAM Role Configuration

For better security, you should use IAM roles instead of hardcoded credentials:

yaml

storage_config:
  aws:
    bucketnames: loki-data
    region: us-west-2
    # No access_key_id or secret_access_key means Loki will use the instance's IAM role
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

Make sure your IAM policy includes these permissions:

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LokiS3Access",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::loki-data",
        "arn:aws:s3:::loki-data/*"
      ]
    }
  ]
}

Google Cloud Storage (GCS) Configuration

Google Cloud Storage offers similar capabilities to S3 and can be configured with Loki as follows:

Basic GCS Setup

yaml

storage_config:
  gcs:
    bucket_name: loki-logs
    chunk_buffer_size: 10485760  # 10MB
    request_timeout: 2m
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: gcs

Authentication Options

GCS supports multiple authentication methods:

Using Service Account JSON

yaml

storage_config:
  gcs:
    bucket_name: loki-logs
    service_account: |
      {
        "type": "service_account",
        "project_id": "your-project-id",
        "private_key_id": "your-private-key-id",
        "private_key": "-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
",
        "client_email": "[email protected]",
        "client_id": "client-id",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/loki-service-account%40your-project-id.iam.gserviceaccount.com"
      }

Using Environment Variables

yaml

storage_config:
  gcs:
    bucket_name: loki-logs
    # Loki will use GOOGLE_APPLICATION_CREDENTIALS environment variable

When deploying to GKE, you can use Workload Identity which is more secure than storing service account keys.

Storage Schema Configuration

The schema_config section defines how Loki organizes data in your object storage. This is critical for performance and cost optimization:

yaml

schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: aws  # or gcs for Google Cloud Storage
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

This configuration tells Loki to:

• Use this schema from July 1, 2020 onwards
• Use boltdb-shipper for the index
• Use AWS S3 for the object store
• Use schema version 11
• Prefix index tables with "loki_index_"
• Create a new index table every 24 hours

Storage Retention Settings

To manage costs and performance, configure retention periods for your log data:

yaml

limits_config:
  retention_period: 744h  # 31 days

compactor:
  working_directory: /loki/compactor
  shared_store: aws  # or gcs
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

Practical Example: Complete Loki Configuration with S3

Let's put everything together in a complete configuration example for a production Loki setup with S3:

yaml

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retention_period: 30s

schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  aws:
    bucketnames: loki-logs
    region: us-west-2
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: aws

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 744h

compactor:
  working_directory: /loki/compactor
  shared_store: aws
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

analytics:
  reporting_enabled: false

Monitoring Object Storage Usage

Once configured, you'll want to monitor your storage usage to optimize costs and performance. Grafana provides useful dashboards for this purpose:

Key metrics to monitor:

• loki_ingester_memory_chunks: Number of chunks in memory
• loki_chunk_store_index_entries_per_chunk: Ratio of index entries to chunks
• cortex_storage_api_response_latency_seconds: Latency of storage operations
• cortex_chunk_store_chunks_stored_total: Total chunks stored

Performance Tuning

Object storage access can become a bottleneck. Consider these tuning options:

1. Caching: Configure appropriate cache sizes in the chunk_cache_config section
2. Chunk size: Adjust chunk_target_size for optimal performance (default 1.5MB)
3. Parallelism: Tune the query parallelism settings
4. Index periods: Shorter index periods mean faster queries but more storage overhead

Common Troubleshooting

Here are solutions to common issues when working with object storage:

Issue	Possible Solution
Slow query performance	Check index and chunk cache configurations
Access denied errors	Verify IAM permissions or credentials
High storage costs	Review retention period and compression settings
Missing data	Check storage class (don't use Glacier or equivalent)
Failed writes	Ensure write permissions and check quotas

Summary

Object storage solutions like Amazon S3 and Google Cloud Storage provide the foundation for a scalable, durable, and cost-effective Grafana Loki deployment. By properly configuring your storage backend, you can ensure:

• Unlimited scaling for log storage
• Separation of compute and storage resources
• Cost optimization through retention policies
• Durable and reliable log storage

Remember that storage configuration choices significantly impact both performance and cost, so take time to design a solution that meets your specific requirements.

Additional Resources

To learn more about Loki storage options, check out:

• Grafana Loki Storage Documentation
• Loki Storage Best Practices
• Understanding BoltDB Shipper

Exercises

1. Set up a local Loki instance using MinIO (S3-compatible storage) for testing
2. Create a Grafana dashboard to monitor your Loki storage metrics
3. Experiment with different retention periods and measure the impact on storage costs
4. Compare query performance between different index.period settings
5. Implement a backup strategy for your Loki object storage