Skip to main content

Dashboard Annotations in Grafana

Introduction​

Annotations in Grafana provide a powerful way to mark points in time on your dashboards with rich contextual information. When working with Loki log data, annotations can help correlate specific events with your log visualization patterns, creating more insightful and actionable dashboards. This guide will teach you how to leverage annotations to enhance your Grafana Loki dashboards.

What Are Annotations?​

Annotations are visual markers displayed as vertical lines or regions on graph panels that represent specific events. They can be:

  • Manual additions to highlight important moments
  • Query-based, automatically generated from your data sources
  • Imported from external systems

Annotations in a Grafana dashboard

Types of Annotations in Grafana​

1. Built-in Annotations​

Grafana comes with several built-in annotation types:

  • Grafana Annotations: Created manually through the Grafana UI
  • API Annotations: Added programmatically via the Grafana HTTP API
  • Dashboard Annotations: Specific to a single dashboard
  • Organization Annotations: Visible across all dashboards in your organization

2. Data Source Annotations​

You can query your data sources (including Loki) to automatically generate annotations based on specific events in your data.

Setting Up Annotations in Grafana​

Creating Manual Annotations​

To add a manual annotation to your dashboard:

  1. Hover over a graph panel and press Ctrl+Click (or Cmd+Click on Mac)
  2. Enter a description for your annotation
  3. Add optional tags to categorize the annotation
  4. Click "Save"

Configuring Annotation Queries with Loki​

Let's create a Loki-based annotation to mark error events:

{app="my-application"} |= "error" | logfmt | level="error"

This query will create annotations for all error-level log entries from "my-application".

Practical Example: Using Annotations with Loki Logs​

Let's walk through a complete example of setting up annotations to mark deployment events alongside application logs.

Step 1: Set Up a Dashboard with Loki Log Visualization​

First, create a dashboard with a Loki panel showing application performance:

javascript
{app="production-app"} | json | status_code != 200

Step 2: Create an Annotation Query for Deployments​

Navigate to Dashboard settings → Annotations and add a new annotation:

  1. Name: "Deployments"
  2. Data source: Loki
  3. Query:
{app="deployment-service"} |= "Deployment started" | line_format "{{.message}}"

Step 3: Configure Annotation Display​

Configure how your annotations will display:

  1. Enable "Show in: All panels"
  2. Select a distinctive color (e.g., vertical green lines)
  3. Set Tags: "deployment", "release"

Step 4: Analyze the Results​

Now your dashboard shows vertical green lines marking deployments alongside your application logs, helping you correlate deployments with any changes in error patterns.

Using Annotations for Incident Analysis​

Annotations are particularly valuable for incident analysis. Here's how to use them effectively:

Example: Annotating an Outage Investigation​

  1. Create a dashboard with Loki queries showing relevant system metrics

  2. Add manual annotations at key investigation points:

    • When the incident was detected
    • When mitigation steps were taken
    • When the system recovered

  3. Add a Loki query-based annotation to show related events:

{app="monitoring"} |= "alert" | label_format message="{{ .level }}: {{ .message }}"

This creates a comprehensive timeline of the incident for post-mortem analysis.

Advanced Annotation Techniques​

Using Templates in Annotations​

You can use templates in your Loki annotation queries to make them more dynamic:

{app="$application"} |= "error" | line_format "Error: {{.message}}"

Where $application is a dashboard variable that users can change.

Using LogQL to Create Meaningful Annotations​

Leverage LogQL's filtering and processing capabilities to create meaningful annotations:

{app="web-server"} 
| json 
| status_code >= 500 
| line_format "HTTP {{.status_code}} error: {{.path}}"

This creates annotations for all HTTP 500 errors, showing the affected path.

Creating Region Annotations​

To highlight time ranges rather than single points, you can use region annotations:

  1. In the Annotations settings, enable "Region" mode
  2. Create a Loki query that returns both a start and end time

Integration with Alert System​

Annotations can automatically mark when alerts were triggered:

  1. Go to Dashboard settings → Annotations
  2. Add a new annotation with "Grafana Alerts" as the data source
  3. Filter by alert name, state, or tags

Best Practices for Annotations​

  1. Use consistent color coding: Assign specific colors to different types of events for quick visual identification
  2. Keep annotations concise: Use short, descriptive text that conveys essential information
  3. Use tags effectively: Create a tagging system for easy filtering
  4. Don't overuse: Too many annotations can make a dashboard cluttered and difficult to read
  5. Combine with dashboard variables: Make annotations dynamic using template variables

Common Issues and Troubleshooting​

Annotations Not Showing Up​

If your Loki-based annotations aren't appearing:

  1. Verify your Loki query returns results within the dashboard's time range
  2. Check that your Loki query includes a timestamp field
  3. Ensure your permissions allow annotation creation

Performance Considerations​

Annotation queries run each time a dashboard loads. For optimal performance:

  1. Limit the scope of your Loki queries with specific label filters
  2. Add time range filters when possible
  3. Consider using pre-computed annotations for frequently accessed dashboards

Summary​

Annotations are a powerful feature in Grafana that can significantly enhance your Loki dashboards by providing contextual information alongside your log data. By using annotations effectively, you can:

  • Correlate system events with log patterns
  • Mark important operational moments
  • Create comprehensive timelines for incident analysis
  • Improve the overall storytelling capabilities of your dashboards

Exercises​

  1. Create a dashboard showing Loki log data and add manual annotations marking three significant events in your system.
  2. Set up a Loki query-based annotation that marks all deployment events.
  3. Create a region annotation showing maintenance windows and observe how your logs change during these periods.
  4. Configure alert annotations and correlate them with error logs from Loki.

Additional Resources​


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)