Observer
Observer

Grafana IRM

Fire and resolve a Grafana IRM alert from Observer via an incoming webhook.

Observer sends incidents to Grafana IRM (Incident Response and Management) through a custom incoming-webhook integration. One Observer incident maps to one Grafana IRM alert.

Set up

  1. Create a Webhook integration

    In Grafana IRM, add an Integration of type Webhook. Grafana issues a unique URL for it, for example https://<stack>.grafana.net/integrations/v1/webhook/<token>/.

  2. Add the integration in Observer

    In the console, open Alerts, choose Add alert, pick Grafana IRM, and paste the integration URL. The URL embeds a token, so it is stored encrypted. Choose org-wide or a single page, then save.

  3. Test it

    Use Test on the row. A test alert should appear in Grafana IRM. Resolve it there when you are done.

Lifecycle mapping

Observer eventGrafana IRM action
Incident publishedFire an alert (grouped by alert id)
Incident update postedFire on the same alert id
Incident resolvedResolve the grouped alert

Correlation is by alert_uid, derived from the Observer incident id (observer-incident-<id>). New events group onto that alert; resolution posts to the integration URL's resolve action for the same id, so the lifecycle produces one alert that fires and then resolves.

Troubleshooting

  • No alert appears: confirm the URL is the full integration webhook URL, including the trailing token segment, and that the integration is enabled in Grafana IRM. A 4xx is surfaced in the Observer delivery result.
  • Resolve does not close the alert: the resolve posts to the same URL with a resolve action. Make sure the integration is a Webhook integration that accepts incoming alerts.
Was this page helpful?