Skip to main content
A metric’s status flips through three layers in order:
  1. Threshold evaluation: the strict-operator rule applied to each pushed value (see threshold operators).
  2. Confirmation time: a status only flips after holding the new state for the configured time.
  3. Snooze (optional): a metric can be snoozed for a window so it does not affect status pages or fire webhook events while operators tune it.
This page covers steps 2 and 3.

​
Why confirmation time exists

A naive implementation would publish every status change the agent reports. In practice, metrics flap. A network blip pushes one bad sample, the next sample is fine, the on-call gets paged twice per minute. Confirmation time requires the new status to hold for a minimum duration before propagating. Configure two values per metric:
  • Confirmation time, getting worse: how long the metric must report the new status before flipping into a worse band (healthy to degraded, or healthy to unhealthy). API and config field: dwell_seconds_to_breach.
  • Confirmation time, recovering: how long the metric must report the new status before flipping into a better band (unhealthy to healthy). API and config field: dwell_seconds_to_recover.
The defaults are conservative: 180 seconds getting worse, 300 seconds recovering. Heartbeat metrics and inbound alert metrics default to 0 for both: a heartbeat already has its grace period, and an alert sender already debounces before it fires. Asymmetric values (“quick to flag, slow to recover”) are appropriate for systems where premature recovery announcements have a higher cost than a delayed unhealthy alert. Both values are editable per metric under Thresholds > Missing data and confirmation time in the metric’s settings, from 0 to 3600 seconds. A confirmation time of 0 flips on the next confirming sample. Configuration-as-code documents accept the same values through the optional dwell_seconds_to_breach and dwell_seconds_to_recover keys on each metric entry, and the API returns them on the metric detail endpoint. Transitions involving no_data or unknown always use the recovering confirmation time, in either direction. A single missed poll never declares a metric down or up. Confirmation runs on the clock. An agent stores a reading when the status changes and on its regular report, not on every poll. So Observer checks pending changes every 15 seconds. Once the confirmation time has passed with no contrary reading, the status flips about a minute later (a short grace for late readings), without waiting for the next report. Late readings don’t page. An agent that was offline sends its buffered readings when it reconnects. They are recorded in history, but an outage that already ended inside that backlog pages no one. If the problem is still happening when the readings arrive, you get one alert.

​
Status sources

Three statuses come from the strict-operator rule:
  • healthy
  • degraded
  • unhealthy
Two outcomes come from the collection layer, not from values:
  • no_data: the agent attempted a probe but produced no value. The reason code is recorded alongside (ECONNREFUSED, ETIMEDOUT, no_data_for_query, etc.; see Reason codes).
  • no_data with reason stale_agent_silent: no push arrived within the staleness cutoff.
Both are operational signals and surface in the agent dashboard. Probe-side no_data fires metric.no_data; staleness does not (see below). Probe-side no_data is not free with respect to SLOs, though: unless the metric’s no-data treatment folds it to healthy or degraded, no_data samples burn under the default down unknown-data policy. A stale metric does not burn (see below). See SLOs and error budgets.

​
Stale data tolerance

Confirmation time handles the small flaps. A separate read-time rule handles a much larger gap: what happens when no sample arrives at all, because the agent is crashed, the network between the agent and Observer Cloud is partitioned, or Observer Cloud itself is degraded. A metric is stale when its last push timestamp is older than a cutoff derived from its push interval: three times the interval, capped at 15 minutes, except that the cap never falls below one minute past the next expected push. The push interval is the metric’s Report at least every setting (interval_agent_push): 5 to 60 minutes, 15 by default, with a plan floor of 10 on Free and Starter.
Worked through: A metric with no push interval set is treated as 15 minutes. The 3× multiplier gives the agent one full retry-and-backoff window before a missing push is considered a problem. The 15-minute cap stops a moderately slow cadence from masking an outage. The push_interval + 1 term keeps a genuinely slow cadence from being declared stale before its next push is even due: a metric that pushes hourly is not blind at 15 minutes. Staleness is enforced on both sides. A sweeper stamps the stored status to no_data with reason stale_agent_silent once the cutoff passes, and the status-page and embed renderers apply the same rule at read time, so a page loaded between sweeps still reads correctly. A stale metric is excluded from the service rollup. It is not counted toward SLO burn. The metric.status_changed and metric.no_data webhooks are not fired on staleness transitions. What does fire: agent.lag_high and agent.offline, which speak to the actual cause. They are operator-facing. When every metric on a service is stale, the service renders as monitoring_delayed with a “Last known: Operational” caption alongside. See Observer availability for the full trust contract.

​
Snooze

A metric can be snoozed until a future timestamp (sometimes called shadow mode). While snoozed:
  • The metric still pushes status to the cloud.
  • Status pages do not consume the snoozed metric in the rolled-up page status.
  • Webhook events for the snoozed metric are suppressed.
  • The metric’s history is still recorded for later inspection.
Use snooze when introducing a new metric, tuning its threshold, or rolling out a new probe runtime. Once the metric behaves as expected, end the snooze and it joins the public status surface.
Strict everywhereThreshold evaluation is strict in both the agent and in the read path that renders status pages. The same value cannot flip status depending on which surface read it. See the threshold operators reference for examples.
For the full picture of how these verdicts become the page badge, the history bars, and the SLO budgets, see How status is calculated.