Local Storage

Introduction

Local storage is one of the fundamental storage options available in Grafana Loki. As the name suggests, local storage refers to using the local filesystem of the machine running Loki to store log data. This approach is straightforward and requires minimal configuration, making it an excellent starting point for beginners or small-scale deployments.

In this guide, we'll explore how local storage works in Loki, when to use it, its advantages and limitations, and how to configure it properly for your log management needs.

Understanding Local Storage in Loki

Grafana Loki's storage system is designed to separate concerns between different types of data:

1. Index - Contains metadata about your logs
2. Chunks - The actual compressed log content

With local storage, both these components are stored directly on the local filesystem where Loki is running.

When to Use Local Storage

Local storage is ideal for:

• Development environments
• Small-scale deployments
• Testing and learning environments
• Single-node Loki installations
• Scenarios with limited log volume

It's generally not recommended for production environments with high availability requirements or large-scale deployments.

Local Storage Configuration

Configuring local storage in Loki is straightforward. Let's look at a basic configuration example:

yaml

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

Let's break down what each of these configuration options means:

BoltDB Shipper Configuration

The boltdb_shipper section configures how Loki handles the index:

• active_index_directory: The location where Loki stores the active index files
• cache_location: Where Loki keeps its cache of index data
• cache_ttl: How long cached index entries remain valid
• shared_store: Specifies the type of storage (in this case, the local filesystem)

Filesystem Configuration

The filesystem section is simpler:

• directory: Specifies where the actual log chunks are stored

Directory Structure and Management

When using local storage, Loki creates a specific directory structure. Here's what you might see:

/loki/
├── chunks/
│   ├── 2023/
│   │   └── ...
│   └── 2024/
│       └── ...
├── index/
│   ├── 00000000-0000-0000-0000-000000000000/
│   └── ...
└── index_cache/
    └── ...

The chunk files are organized by year and then further subdivided, while index files use UUID-based directories.

Practical Example: Setting Up Local Storage

Let's walk through setting up a Loki instance with local storage:

1. First, create a configuration file named loki-local-config.yaml:

yaml

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

2. Create the required directories:

bash

mkdir -p /loki/chunks /loki/index /loki/index_cache

3. Run Loki with this configuration:

bash

loki -config.file=loki-local-config.yaml

Monitoring Local Storage

When using local storage, it's important to monitor disk usage to avoid running out of space. You can use the built-in metrics that Loki exposes:

• loki_ingester_memory_chunks: Number of chunks in memory
• loki_chunk_store_index_entries_per_chunk: Number of index entries per chunk
• loki_chunk_operations_total: Total chunk operations

You can visualize these metrics using Grafana to keep an eye on your storage usage.

Performance Considerations

Local storage has some performance implications to be aware of:

1. I/O Bound - Performance is largely dependent on the disk I/O capabilities of your system
2. Scaling Limitations - Cannot scale horizontally as easily as cloud storage options
3. Backup Challenges - Manual backup processes are required

For better performance:

• Use SSDs instead of HDDs when possible
• Ensure adequate free disk space (at least 30% free)
• Monitor disk I/O and adjust retention periods as needed

High Availability Considerations

Local storage is not ideal for high availability setups because:

1. If the node goes down, the logs become inaccessible
2. No built-in replication across multiple nodes
3. Difficult to load balance between multiple instances

If you need high availability, consider using object storage solutions like S3, GCS, or Azure Blob Storage.

Comparing Local Storage vs. Object Storage

Let's compare local storage with object storage options:

Feature	Local Storage	Object Storage (S3, GCS, etc.)
Setup complexity	Low	Medium to High
Cost	Low (just disk space)	Varies based on usage
Scalability	Limited	High
High Availability	No	Yes
Performance	Depends on disk I/O	Depends on network I/O
Backup/Recovery	Manual	Often built-in
Use case	Development, small deployments	Production, large deployments

Data Retention and Cleanup

Local storage requires manual management of data retention. Loki provides configuration options to help:

yaml

limits_config:
  retention_period: 720h  # 30 days

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

This configuration enables the compactor to delete data older than the specified retention period.

Troubleshooting Local Storage Issues

Common issues with local storage and how to resolve them:

1. Disk Space Exhaustion

   • Symptom: Loki stops ingesting logs
   • Solution: Increase disk space or reduce retention period

2. Slow Query Performance

   • Symptom: Queries take a long time to complete
   • Solution: Check disk I/O, consider moving to SSD storage

3. File Permission Issues

   • Symptom: Loki fails to start with permission errors
   • Solution: Ensure the Loki process has write permissions to the storage directories

Migration Path

As your deployment grows, you might need to migrate from local storage to object storage. Here's a simple migration path:

1. Set up object storage (S3, GCS, etc.)
2. Configure Loki to use the new object storage
3. Use Loki's built-in replication to copy data (if available)
4. Alternatively, use manual backup and restore procedures

yaml

# Example configuration after migration to S3
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
    shared_store: s3
  aws:
    s3: s3://access_key:secret_access_key@region/bucket_name
    s3forcepathstyle: true

Local Storage in Docker Environments

When running Loki in Docker, you'll need to mount volumes for local storage:

bash

docker run -d --name loki \
  -v $(pwd)/loki:/loki \
  -p 3100:3100 \
  grafana/loki:latest \
  -config.file=/etc/loki/local-config.yaml

Make sure your configuration file points to the correct paths inside the container.

Local Storage in Kubernetes

For Kubernetes deployments, you'll need to use Persistent Volumes:

yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: loki-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
spec:
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
        - name: loki
          image: grafana/loki:latest
          ports:
            - containerPort: 3100
          volumeMounts:
            - name: config
              mountPath: /etc/loki
            - name: storage
              mountPath: /loki
      volumes:
        - name: config
          configMap:
            name: loki-config
        - name: storage
          persistentVolumeClaim:
            claimName: loki-storage

Summary

Local storage in Grafana Loki provides a simple, low-overhead way to store logs for development, testing, and small-scale deployments. While it lacks the high availability and scalability features of object storage solutions, it's perfect for getting started with Loki and understanding how the system works.

Key takeaways:

• Local storage uses the local filesystem to store both index and chunk data
• Configuration is straightforward, requiring minimal setup
• Best suited for development and small-scale deployments
• Performance depends largely on the underlying disk I/O capabilities
• Requires manual management of disk space and backups
• Provides a stepping stone before migrating to object storage solutions

Additional Resources

• Grafana Loki Documentation
• Loki Storage Configuration Guide
• Loki Retention and Compaction

Exercises

1. Set up a local Loki instance using local storage and send some logs to it using Promtail or LogQL.
2. Monitor the disk usage of your local storage and observe how it grows over time.
3. Experiment with different retention periods and observe their impact on storage usage.
4. Create a backup strategy for your local storage setup.
5. Try configuring Loki with both local storage for chunks and object storage for the index.