Webhook deliveries failing

Q: Step 1: read the delivery log

Open **Webhooks**, the subscription in question, then **Delivery log**. Each entry shows: - event_type and event_id. - attempted_at. - response_status (or a network-level error string). - response_body (truncated). If the log is empty, the events the subscription is bound to have not fired since the subscription was created. Trigger a test event by changing a metric's threshold to flip status, or wait for an organic event.

Q: Step 2: non-2xx from the receiver

| Status | Probable cause | |---|---| | 400 | The receiver expects a different payload schema. Compare against Webhook payload reference (/docs/reference/webhook-payloads). | | 401 / 403 | Authentication required. Receivers like generic Slack apps or HMAC-protected endpoints require headers Observer does not set by default. | | 404 | URL is wrong. Re-paste from the receiver's documentation. | | 429 | Receiver is rate-limiting. Reduce subscription scope or contact the receiver's vendor. | | 5xx | Receiver is failing. The delivery worker retries with exponential backoff up to a fixed cap; deliveries are then moved to the dead-letter view. | The delivery worker retries non-2xx responses. If retries exhaust, the entry moves to the **Dead letter** view; manual replay is available there.

Q: Step 3: network errors

If response_status is missing and the log shows a network-level error: | Error substring | Probable cause | |---|---| | ECONNREFUSED | Receiver host is not listening on the configured port. | | ENOTFOUND | DNS resolution failed. Verify the URL hostname. | | ETIMEDOUT | Receiver did not respond within the request timeout. | | CERT_HAS_EXPIRED / UNABLE_TO_VERIFY_LEAF_SIGNATURE | The receiver's TLS certificate is expired or untrusted by the public CA bundle. Renew the certificate; Observer does not accept self-signed certificates against a public endpoint. |

Q: Step 5: subscription is disabled

A subscription whose **enabled** flag is off does not deliver events. The subscription edit page exposes the toggle. If a subscription was disabled while debugging, re-enable it to resume deliveries; new events fire from the moment it is re-enabled, not backfilled. Webhook delivery actions write audit log rows under the webhook category. Audit log rows carry the failing receiver URL and the response status, which helps when correlating across multiple subscriptions.

Diagnose a webhook subscription whose deliveries do not reach the receiver, or whose receiver rejects them.

A failing webhook subscription presents in one of three ways: the delivery log shows non-2xx responses from the receiver, the log shows network errors before the receiver was reached, or the log is empty when the operator expected events.

Step 1: read the delivery log

Open Webhooks, the subscription in question, then Delivery log. Each entry shows:

event_type and event_id.
attempted_at.
response_status (or a network-level error string).
response_body (truncated).

If the log is empty, the events the subscription is bound to have not fired since the subscription was created. Trigger a test event by changing a metric's threshold to flip status, or wait for an organic event.

Step 2: non-2xx from the receiver

Status	Probable cause
`400`	The receiver expects a different payload schema. Compare against Webhook payload reference.
`401` / `403`	Authentication required. Receivers like generic Slack apps or HMAC-protected endpoints require headers Observer does not set by default.
`404`	URL is wrong. Re-paste from the receiver's documentation.
`429`	Receiver is rate-limiting. Reduce subscription scope or contact the receiver's vendor.
`5xx`	Receiver is failing. The delivery worker retries with exponential backoff up to a fixed cap; deliveries are then moved to the dead-letter view.

The delivery worker retries non-2xx responses. If retries exhaust, the entry moves to the Dead letter view; manual replay is available there.

Step 3: network errors

If response_status is missing and the log shows a network-level error:

Error substring	Probable cause
`ECONNREFUSED`	Receiver host is not listening on the configured port.
`ENOTFOUND`	DNS resolution failed. Verify the URL hostname.
`ETIMEDOUT`	Receiver did not respond within the request timeout.
`CERT_HAS_EXPIRED` / `UNABLE_TO_VERIFY_LEAF_SIGNATURE`	The receiver's TLS certificate is expired or untrusted by the public CA bundle. Renew the certificate; Observer does not accept self-signed certificates against a public endpoint.

Step 4: signature verification on the receiver

If the receiver computes a signature mismatch but the URL and secret are correct:

Confirm the secret is the value Observer's webhook subscription page shows, not a copy with whitespace.
Confirm the receiver computes HMAC-SHA-256(secret, raw_body) and compares against the X-Observer-Signature header value (after the sha256= prefix).
Confirm the receiver hashes the raw request body, not a re-serialised JSON. Some web frameworks parse and re-serialise request bodies on the way to the handler; the recomputed signature does not match.

Step 5: subscription is disabled

A subscription whose enabled flag is off does not deliver events. The subscription edit page exposes the toggle. If a subscription was disabled while debugging, re-enable it to resume deliveries; new events fire from the moment it is re-enabled, not backfilled.