Webhook deliveries failing

Webhooks are configured on the Alerts page as destinations (the Generic webhook channel, or a Slack, Discord, or Teams webhook). A failing webhook presents in one of three ways: deliveries show non-2xx responses from the receiver, deliveries fail before the receiver was reached, or nothing is delivered when you expected an alert.

​

Step 1: check the destination row

Open Alerts in the console. Each row under Destinations shows:

• Status: the outcome of the most recent attempt (Delivered, Retrying, Failed, or Skipped), or Paused when the switch is off.
• Last delivery: when the most recent attempt happened, or Never sent if the destination has not been sent anything yet.
• Delivered (7d): the share of deliveries that succeeded in the last 7 days. Select it to filter Latest alerts to that destination.

To send a sample alert now, use Send test from the row’s actions menu.

​

Step 2: read Latest alerts

Latest alerts, at the top of the page above Destinations, shows the latest attempts across every destination. Use the Failed filter to see only failures. Each row shows the status, channel, destination, event, a detail column with the HTTP status or error, and when it happened. Open a row to see the attempt number, the full error, and the payload that was sent. A failed delivery with a stored payload has a Redeliver button. Use it after fixing the receiver to send the same payload again.

​

Step 3: non-2xx from the receiver

The detail column shows HTTP <status> · http_<status>, followed by the start of the receiver’s response body when it returned one. 429, 5xx, and network errors are retried with backoff, up to 5 attempts. Other 4xx responses fail immediately and are not retried. While a delivery is being retried its status is Retrying; once attempts run out it is Failed.

​

Step 4: errors before the receiver

Observer does not follow redirects. A URL that answers with a 3xx is reported as a failed delivery; use the final URL instead.

​

Step 5: signature verification on the receiver

If the receiver reports a signature mismatch but the URL and secret are correct:

• Confirm the receiver uses the same signing secret you entered in the destination, without surrounding whitespace. The secret is not shown again after saving; type a new one into the destination and update the receiver if unsure.
• Confirm the receiver computes HMAC-SHA-256(secret, X-Observer-Timestamp + "." + raw_body) and compares it against X-Observer-Signature (after sha256=).
• Confirm the receiver hashes the raw request body, not a re-serialized JSON. Some web frameworks parse and re-serialize request bodies on the way to the handler; the recomputed signature then does not match.

​

Step 6: nothing is delivered

If Latest alerts shows nothing for the destination when you expected an alert:

• Enabled: the switch on the row must be on. A paused destination delivers nothing, and events that happen while it is off are not delivered later.
• Snooze: a snoozed row shows when it wakes in its actions menu. Use Clear snooze to resume.
• Alerts and routing: open the destination and confirm the alert is selected under Alerts (the Paging preset sends only incidents and metrics turning unhealthy), and that the Routing filters (services, metrics, minimum severity) let it through.
• Scope: a destination set to one page under Routing > Status page only receives alerts for that page. Agent health, SLO, and SLA Ledger alerts go only to destinations set to All status pages (organization-wide).
• Daily digest: with the daily digest on, alerts are batched into one delivery per day instead of being sent immediately.
• Drafts: draft incidents never send alerts. Alerts start when the incident is published.

​

Legacy webhook subscriptions

Webhook subscriptions created before Alerts existed keep delivering, but they no longer have a console page. To inspect or change one, add the endpoint as a Generic webhook on the Alerts page.