Skip to main content

Alert Notifications

Introduction​

Alert notifications are a crucial component of Grafana's alerting system that determine how users are informed when alerts are triggered. They serve as the bridge between alert detection and human response, ensuring that the right people receive timely information about system or application issues that require attention.

In this guide, you'll learn how Grafana handles alert notifications, how to configure different notification channels, and best practices for setting up effective alerting workflows that help your team respond quickly to incidents.

Understanding Alert Notifications in Grafana​

Alert notifications in Grafana are managed through notification policies and contact points. These two components work together to determine:

  1. Who receives alert notifications
  2. How they receive them (email, Slack, webhook, etc.)
  3. When notifications are sent (immediately, with a delay, or during specific time windows)
  4. What information is included in the notifications

Let's break down these key components:

Contact Points​

Contact points define the specific methods or channels through which alert notifications are delivered. Grafana supports numerous notification channels including:

  • Email
  • Slack
  • Microsoft Teams
  • Discord
  • PagerDuty
  • OpsGenie
  • Webhook
  • And many more

Each contact point can be configured with specific settings relevant to that notification method.

Notification Policies​

Notification policies define the rules for routing alerts to specific contact points. They allow you to:

  • Group similar alerts together
  • Apply timing rules (when to notify)
  • Add custom templates for notification content
  • Route different types of alerts to different contact points based on labels

Setting Up Alert Notifications​

Let's walk through the process of configuring alert notifications in Grafana:

1. Creating Contact Points​

To create a new contact point:

  1. Navigate to Alerting in the Grafana menu
  2. Select Contact points
  3. Click + Add contact point
  4. Choose a notification integration type (e.g., Email, Slack)
  5. Configure the integration-specific settings

Here's an example of configuring an email contact point:

js
// Email integration configuration example
{
  "name": "DevOps Team Email",
  "type": "email",
  "settings": {
    "addresses": "[email protected]",
    "singleEmail": true
  }
}

And here's how you would configure a Slack contact point:

js
// Slack integration configuration example
{
  "name": "Critical Alerts Slack",
  "type": "slack",
  "settings": {
    "recipient": "#critical-alerts",
    "token": "xoxb-your-slack-token",
    "username": "Grafana Alerting",
    "icon_emoji": ":rotating_light:",
    "mentionChannel": "here"
  }
}

2. Defining Notification Policies​

After setting up contact points, you'll want to create notification policies to route alerts:

  1. Go to Alerting > Notification policies
  2. Configure the Default policy or create new specific policies
  3. Define matching criteria using alert labels
  4. Set the contact point to use for matching alerts
  5. Configure timing options (grouping, timing, etc.)

Here's a visualization of how notification policies work:

Notification Templates​

Grafana allows you to customize the content and format of alert notifications using templates. Templates can include alert details, metric values, links to dashboards, and custom messaging.

Basic Template Example​

Here's a simple notification template for Slack:

js
// Slack message template
{
  "text": "{{ if eq .Status \"firing\" }}🔥{{ else }}✅{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}

  *Summary:* {{ .CommonAnnotations.summary }}

  *Description:* {{ .CommonAnnotations.description }}

  *Severity:* {{ .CommonLabels.severity }}

  {{ if ne .SilenceURL \"\" }}*Silence:* {{ .SilenceURL }}{{ end }}
  {{ if ne .DashboardURL \"\" }}*Dashboard:* {{ .DashboardURL }}{{ end }}"
}

This template creates a formatted message with:

  • An emoji indicating alert status (firing or resolved)
  • Alert name in uppercase
  • Summary and description from alert annotations
  • Severity from alert labels
  • Links to silence the alert or view the dashboard (if available)

Practical Examples​

Let's explore some real-world notification setups for different scenarios:

Example 1: Multi-Tier Alert Routing​

For a production environment, you might want different notification channels based on alert severity:

  1. Critical alerts: Send to PagerDuty for immediate action
  2. Warning alerts: Send to Slack channel for awareness
  3. Info alerts: Log to a monitoring system without human notification

Here's how to configure this with notification policies:

js
// Critical alerts policy
{
  "name": "Critical Alerts",
  "group_by": ["alertname", "job"],
  "contact_point": "pagerduty-oncall",
  "matchers": [
    {
      "name": "severity",
      "value": "critical",
      "type": "="
    }
  ]
}

// Warning alerts policy
{
  "name": "Warning Alerts",
  "group_by": ["alertname", "job"],
  "contact_point": "team-slack",
  "group_wait": "30s",
  "group_interval": "5m",
  "matchers": [
    {
      "name": "severity",
      "value": "warning",
      "type": "="
    }
  ]
}

// Info alerts policy
{
  "name": "Info Alerts",
  "group_by": ["alertname"],
  "contact_point": "monitoring-system",
  "matchers": [
    {
      "name": "severity",
      "value": "info",
      "type": "="
    }
  ]
}

Example 2: Business Hours vs. After Hours​

You might want different notification behaviors during business hours versus after hours:

js
// Business hours policy
{
  "name": "Business Hours",
  "contact_point": "email-slack-combo",
  "time_intervals": ["business-hours"],
  "group_by": ["alertname", "job"]
}

// After hours policy
{
  "name": "After Hours",
  "contact_point": "pagerduty-oncall",
  "time_intervals": ["after-hours"],
  "group_by": ["alertname"]
}

With corresponding time intervals defined:

yaml
# Time interval definitions
business-hours:
  time_zone: "America/New_York"
  weekdays: ["monday:friday"]
  times:
    - start_time: "09:00"
      end_time: "17:00"

after-hours:
  time_zone: "America/New_York"
  weekdays: ["monday:friday"]
  times:
    - start_time: "17:00"
      end_time: "09:00"
  weekend: true

Best Practices for Alert Notifications​

1. Avoid Alert Fatigue​

  • Group related alerts to reduce notification volume
  • Use muting timings during maintenance windows
  • Implement proper severity levels and only page for critical issues

2. Provide Actionable Information​

  • Include clear descriptions of what triggered the alert
  • Add links to relevant dashboards or runbooks
  • Include metric values and thresholds that were violated

3. Set Up Escalation Paths​

  • Configure notification retries for unanswered critical alerts
  • Use services like PagerDuty or OpsGenie for escalation chains
  • Have backup contact points for different scenarios

4. Test Your Notifications​

Regularly test your alert notification setup:

js
// Test alert configuration
{
  "name": "Test Alert",
  "condition": "B",
  "data": [
    {
      "refId": "A",
      "datasource": {
        "uid": "-- Grafana --",
        "type": "datasource"
      },
      "queryType": "randomWalk"
    },
    {
      "refId": "B",
      "datasource": {
        "uid": "-- Grafana --",
        "type": "datasource"
      },
      "queryType": "testAlert"
    }
  ],
  "noDataState": "OK",
  "execErrState": "OK",
  "notifications": []
}

Troubleshooting Notification Issues​

If your alert notifications aren't working as expected, check these common issues:

  1. Integration configuration: Verify API keys, tokens, and endpoint URLs
  2. Network connectivity: Ensure Grafana can reach notification services
  3. Rate limiting: Some services (like Slack) have API rate limits
  4. Template errors: Check for syntax errors in custom templates
  5. Policy matching: Confirm alert labels match your notification policies

Summary​

Alert notifications are the critical last step in your monitoring workflow, ensuring that the right people are informed when incidents occur. In this guide, you've learned:

  • The core components of Grafana's notification system: contact points and notification policies
  • How to configure various notification channels like email, Slack, and PagerDuty
  • How to use templates to customize notification content
  • Best practices for implementing effective notification strategies
  • Troubleshooting common notification issues

By thoughtfully designing your alert notification setup, you can ensure that your team responds quickly to problems while avoiding alert fatigue.

Additional Resources​

Exercises​

  1. Set up an email contact point and test it with a simple alert rule
  2. Create a notification policy that routes different alerts based on their severity
  3. Design a custom template for Slack notifications that includes metric values and dashboard links
  4. Configure a notification setup with different behaviors for business hours versus after hours
  5. Create a multi-tier notification strategy for a hypothetical application with critical, warning, and info alerts

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)