Log Filtering Basics

Introduction

When working with logs in Grafana Loki, one of the most fundamental skills is effectively filtering your log data to find exactly what you need. Log filtering is the foundation of LogQL, Loki's query language, and mastering these basics will help you navigate through vast amounts of log data with precision and efficiency.

In this guide, we'll explore the fundamental concepts of log filtering in Loki, learn the syntax for creating basic filters, and see how these filters can be applied in real-world scenarios. By the end, you'll be able to write simple yet powerful log queries to extract valuable insights from your log data.

What is Log Filtering?

Log filtering is the process of narrowing down log entries based on specific criteria. In Loki, filtering operates on two primary levels:

1. Label filtering - Selecting log streams based on their metadata labels
2. Log content filtering - Searching through the actual content of log lines

Let's explore each of these in detail.

Label Filtering

Labels in Loki are key-value pairs that describe the source of your logs. They are similar to labels in Prometheus and provide a way to organize and categorize your log data.

Basic Label Syntax

The basic syntax for filtering by labels is:

{label="value"}

For example, to query logs from a specific application:

{app="frontend"}

This will return all logs that have the label app with the value frontend.

Multiple Label Matchers

You can combine multiple label matchers to narrow down your query:

{app="frontend", environment="production"}

This query returns logs that have both app="frontend" AND environment="production".

Label Matching Operators

Loki supports several operators for matching labels:

Operator	Description	Example
=	Exactly equal	{app="frontend"}
!=	Not equal	{app="frontend", environment!="testing"}
=~	Regex match	{app=~"front.*"}
!~	Regex not match	{app!~"back.*"}

Label Matching Examples

Let's look at some practical examples:

Example 1: Find all logs from the production environment:

{environment="production"}

Example 2: Find logs from any service except the database:

{service!="database"}

Example 3: Find logs from any service that starts with "auth":

{service=~"auth.*"}

Example 4: Find logs from production frontend or backend services:

{environment="production", service=~"(frontend|backend)"}

Log Content Filtering

Once you've selected log streams using label filters, you can further refine your query by searching for specific text within the log content.

Basic Log Content Filter

To filter log content, use the pipe (|) character followed by filtering expressions:

{label="value"} |= "search string"

Log Line Filter Operators

Loki provides several operators for filtering log content:

Operator	Description	Case-Sensitive	Example
`	=`	Line contains string	Yes | {app="frontend"} |= "error"
!=	Line does not contain string	Yes	{app="frontend"} != "debug"
`	~`	Line matches regex	Yes | {app="frontend"} |~ "error.*timeout"
!~	Line does not match regex	Yes	{app="frontend"} !~ "debug.*info"

Case-Insensitive Filtering

For case-insensitive matching, you can use the (?i) flag in your regex:

{app="frontend"} |~ "(?i)error"

This will match "error", "ERROR", "Error", etc.

Combining Multiple Filters

You can chain multiple content filters to create more complex queries:

{app="frontend"} |= "error" != "timeout" |= "database"

This query will:

1. Select logs from the frontend app
2. Filter for lines containing "error"
3. Exclude lines containing "timeout"
4. Further filter for lines containing "database"

Practical Examples

Let's look at some real-world examples of log filtering:

Example 1: Finding Authentication Failures

{app="auth-service"} |= "authentication failed" |= "user"

This query selects logs from the authentication service and filters for lines containing both "authentication failed" and "user".

Sample input logs:

2023-05-10T12:34:56Z INFO authentication successful for user [email protected]
2023-05-10T12:35:12Z ERROR authentication failed for user [email protected]: invalid password
2023-05-10T12:36:05Z WARN rate limit exceeded for IP 192.168.1.42
2023-05-10T12:36:30Z ERROR authentication failed for user guest: account locked

Query output:

2023-05-10T12:35:12Z ERROR authentication failed for user [email protected]: invalid password
2023-05-10T12:36:30Z ERROR authentication failed for user guest: account locked

Example 2: Monitoring HTTP 5xx Errors

{service="api-gateway"} |~ "HTTP (5\\d{2})"

This query uses a regular expression to find all HTTP 500-level errors in the API gateway logs.

Sample input logs:

2023-05-10T13:01:22Z INFO HTTP 200 GET /api/users 192.168.1.101 response_time=45ms
2023-05-10T13:01:35Z WARN HTTP 404 GET /api/nonexistent 192.168.1.102 response_time=12ms
2023-05-10T13:01:47Z ERROR HTTP 500 POST /api/orders 192.168.1.103 response_time=2543ms
2023-05-10T13:01:59Z ERROR HTTP 503 GET /api/products 192.168.1.104 response_time=5012ms

Query output:

2023-05-10T13:01:47Z ERROR HTTP 500 POST /api/orders 192.168.1.103 response_time=2543ms
2023-05-10T13:01:59Z ERROR HTTP 503 GET /api/products 192.168.1.104 response_time=5012ms

Example 3: Finding Logs for a Specific User Session

{app=~".*-service"} |= "session_id=abc123"

This query searches for a specific session ID across all services.

Filtering Flow Visualization

The following diagram illustrates how label filtering and log content filtering work together:

Performance Considerations

When filtering logs in Loki, keep these performance tips in mind:

1. Labels first, content second: Label filters are processed before log content filters and are much more efficient.
2. Be specific with labels: The more specific your label query, the less data Loki needs to process.
3. Use time ranges: Always specify a time range to limit the amount of data being processed.
4. Avoid regex when possible: Regex operations are more CPU-intensive than simple string matching.

Summary

In this guide, we've covered the fundamentals of log filtering in Grafana Loki using LogQL:

• Label filtering allows you to select specific log streams based on metadata
• Log content filtering lets you search within log lines for specific text or patterns
• You can combine multiple filters to create powerful, targeted queries
• Understanding the performance implications of different filter types helps you write efficient queries

By mastering these basic filtering techniques, you're now ready to effectively navigate through your log data and extract valuable insights.

Exercises

To practice your log filtering skills, try these exercises:

1. Write a query to find all ERROR level logs from production services.
2. Create a filter to find all logs containing "connection refused" but not related to database services.
3. Write a query to find all logs from authentication services where a user has failed to log in more than 3 times.
4. Find all logs containing IP addresses using a regular expression.

Additional Resources

• Grafana Loki Documentation
• LogQL Query Language Reference
• Advanced Log Filtering Techniques