PlansInbound alert sources are available from the Starter plan. Starter
includes 3 sources, Pro 10, Business 25, and Enterprise has no limit. Only
enabled sources count.
How it works
- Your alerting tool sends a webhook to the source’s intake URL with the source’s secret.
- Observer parses the alerts and keeps the current state of each one (firing or resolved).
- Each firing alert is checked against the source’s mapping rules, in order. The first rule that matches decides which metric it drives and with which status.
- Each alert-backed metric takes the worst status among the alerts routed to it. With none firing, it is healthy.
Add a source
1
Create the source
In the console, open Alerts and find the Inbound section. Add a
source, pick the sender (Prometheus Alertmanager, Grafana Alerting,
Datadog, AWS CloudWatch, or Generic JSON), and name it.For Grafana and Generic JSON you can also turn on request signing. See
Signed requests.
2
Copy the URL and the secret
Observer shows the intake URL and the secret. The secret (and the signing
secret, if you turned signing on) is shown once. Copy it now. Observer
stores it hashed or encrypted and cannot show it again. If it leaks, rotate
it from the source page; the old secret stops working immediately.
3
Configure the sender
The source page shows a setup snippet for the sender you picked, with the
URL filled in. The sections below show each one.
4
Add mapping rules
Under Mapping rules, add a rule, set its conditions, and choose the
metric it drives. Pick an existing alert-backed metric or create one from
the rule with New alert-backed metric. See
Mapping rules.
5
Send a test event
Send a test event from the sender (see
Send a test event). It arms the source, and the
source page lists it under Deliveries with the outcome Test. Real
alerts appear under Alerts as they arrive.
Set up each sender
The intake URL has the formhttps://use.observer/api/intake/<source_id>.
The id in the URL only locates the source and authorizes nothing on its
own. Every request must also carry the secret.
- Alertmanager
- Grafana Alerting
- Datadog
- CloudWatch (SNS)
- Generic JSON
Add a webhook receiver to Keep
alertmanager.yml and route the alerts you want
to it. Observer reads webhook payload version 4.alertmanager.yml
send_resolved: true so recoveries reach Observer. Labels become the
alert’s labels, and severity (or priority) becomes its severity. When
an alert has no fingerprint, Observer derives one from its labels.Generic JSON schema
The body is an object with analerts array, or a single alert object on
its own.
Arming
A source is armed by its first verified event: a request that passed the secret check (and the signature check, when signing is on) and could be parsed. A test event or a real alert both arm it. Requests with a wrong secret, a bad signature or a body that cannot be parsed never arm a source.- Before it is armed, the metrics the source drives read No data. Observer has no evidence yet that the sender can reach it, so it does not assume healthy.
- Once it is armed, a metric with no firing alerts routed to it reads healthy.
Send a test event
A test event arms the source and is recorded under Deliveries with the outcome Test. It is never routed to a metric and never changes a status. For every sender except CloudWatch, a request with the headerX-Observer-Test: 1 is a test event. The header is checked after the
secret and the signature, so a test event proves the sender’s
credentials.
- Alertmanager
- Grafana Alerting
- Datadog
- CloudWatch (SNS)
- Generic JSON
Alertmanager has no test button. Run this from a host that can reach
Observer, or wait for the first real notification. Either one arms the
source. With a heartbeat configured, the first
Watchdog notification arms it too.
Heartbeat alert
Without a heartbeat, a source that goes silent looks the same as a source with nothing firing: its metrics read healthy. To tell the two apart, configure an optional heartbeat alert on the source: an alert your sender keeps firing at all times, such as Alertmanager’sWatchdog.
Under Heartbeat on the source page, turn it on and set:
- Heartbeat alert: one or more conditions that identify it, written
like mapping rule conditions. The default is
alertname=Watchdog. - No data after: how many minutes without it before the source’s metrics read No data. The default is 10 minutes, and it can be set from 2 to 1440.
- The heartbeat is never routed as an alert. It never drives a metric, and a firing heartbeat only records that the sender is alive.
- When no heartbeat has arrived within the window, the metrics the source’s enabled rules drive read No data, with the reason “Heartbeat alert missing”. This holds even if other alerts still arrive.
- When the heartbeat resumes, the metrics return to the status their alerts give them.
- Saving the heartbeat settings restarts the window, so a new heartbeat gets one full window to arrive. Turning the heartbeat off returns the metrics to the status their alerts give them.
Authentication
Every request must carry the source’s secret (it starts withobs_in_).
Observer accepts it from, in order:
Authorization: Bearer <secret>Authorization: Basic ...with the secret as the password (any user name)X-Observer-Token: <secret>, for senders that cannot setAuthorization
401. Observer stores the secret hashed and
never stores the body of a request that failed authentication.
Signed requests
For Grafana and Generic JSON sources you can also require an HMAC-SHA256 signature over the raw request body. Unsigned requests, wrong signatures and timestamps more than 5 minutes from now are refused.- Grafana: set the contact point’s HMAC signature secret to the
source’s signing secret. Grafana sends
X-Grafana-Alerting-Signature. If you also configure the timestamp headerX-Grafana-Alerting-Signature-Timestamp, the signature covers<timestamp>:<body>. - Generic JSON: send
X-Observer-Timestamp(Unix seconds or milliseconds) andX-Observer-Signature: sha256=<hex(HMAC-SHA256(signing_secret, timestamp + "." + body))>. This is the same format Observer uses to sign its own outbound webhooks.
CloudWatch signatures
For CloudWatch sources, Observer verifies the AWS signature on every SNS message (signature versions 1 and 2) and only fetches signing certificates from Amazon SNS hosts. The secret in the subscription URL is still required.Mapping rules
Rules are evaluated in order, and the first enabled rule whose conditions all match wins. A rule with no conditions matches every alert. Alerts that match no rule are kept on the source page but drive nothing. Conditions are Prometheus style and test one label each:
Regexes match the whole value, as in Alertmanager. Patterns longer than
200 characters, nested repeats such as
(a+)+, repeated alternations such
as (a|aa)*, backreferences and lookarounds are refused.
Each rule sets the status its alerts drive while firing:
- Unhealthy or Degraded: every matching alert drives that status.
- By severity: the alert’s severity decides. You can set overrides on the rule. Without an override, these defaults apply:
An alert whose severity Observer does not recognize is treated as
unhealthy, so it never reads healthier than it is.
Editing rules re-routes the source’s alerts that are firing at that time.
Firing and resolved
- An alert-backed metric is unhealthy if any routed alert maps to unhealthy, degraded if the worst maps to degraded, and healthy when no routed alert is firing.
- Its value is a severity score:
0healthy,1degraded,2unhealthy. The thresholds are fixed to match, so the value and the status always agree. - Dwell starts at 0, because senders already debounce (for example
Alertmanager’s
for:). You can raise it on the metric. Status changes then follow the same dwell, incident, SLO and notification rules as any other metric. See Thresholds and dwell. - Late messages are ignored: a delayed firing for an alert that has since resolved, or a resolve for an older firing, does not change the state.
- A new alert-backed metric reads No data until its source is armed. It never starts out healthy by assumption.
Auto-resolve
Auto-resolve is off by default. With it off, a firing alert stays firing in Observer until the sender sends a resolved message for it, however long that takes. Observer never resolves an alert on its own. Most senders send a resolved message when an alert clears. If yours does not (for example Alertmanager withsend_resolved: false, Grafana with
Disable resolved message on, or a script that only reports firing), or
if a resolved message is lost, the metric stays unhealthy or degraded
until you resolve it from the sender. For such senders, set
Auto-resolve on the source (from 5 minutes to 7 days): a firing alert
the sender has not repeated within that time is resolved.
Responses
Raw payloads and retention
The source page lists each authenticated delivery under Deliveries, and you can open one to see its raw payload. Raw payloads are kept for 90 days. Resolved alerts are removed 90 days after they resolve. Pausing a source makes it answer403 and change nothing. Deleting a
source stops its URL immediately and removes its rules, alerts and
delivery log. Metrics it drove recover.
Every change to a source, its rules or its secrets is recorded in the
audit log.
