Observer
Observer

How status is calculated

Every number and color on a status page, where it comes from, and why it works that way.

A status page shows several kinds of truth at once: what is happening right now, what happened over the last month, and what you have promised. They are computed differently on purpose. This page walks through each one.

The headline badge#

The badge at the top ("All systems operational") is the page's current state. It is built in two steps.

Step 1: roll up the metrics. Every metric bound to the page has a current status. The page takes the worst one. Severity is ordered: unhealthy, then degraded, then healthy, then unknown. One unhealthy metric outweighs any number of healthy ones. A page is only as healthy as its worst component, so averaging would hide problems.

Step 2: fold in incidents and maintenance. An open incident or a running maintenance window can make the headline worse than the metrics say, never better. If every metric is green but you have published a Sev-1 incident, the headline shows the incident. The rationale: an operator who declares an incident knows something the probes do not yet see, and visitors should hear the human first.

The same escalated verdict is used everywhere the overall status appears: the main page, the incident history page, the embed widget, the badge image, the JSON API, and the feeds. One calculation, many surfaces, so no two views of your page can disagree.

Per-metric status#

Each check produces a value, and the value is compared against your thresholds:

  • If it matches the healthy rule, the metric is healthy.
  • Otherwise, if it matches the unhealthy rule, it is unhealthy.
  • Otherwise it is degraded, the space between your two lines.

Comparisons are strict. "Over 99" means greater than 99, not greater than or equal. A value exactly on the line does not match the rule. This sounds pedantic, but it guarantees the same value can never flip between two statuses depending on which part of the system evaluated it.

The comparison happens on the agent, inside your network. The cloud receives the verdict, not the raw measurement, and never re-decides.

Dwell: why the page does not flicker#

A single bad probe does not change the page. When a metric's status would change, the new status has to hold for a configured dwell window before the page flips. Getting worse and getting better can have different windows, so you can flip to unhealthy quickly but demand a longer stretch of good readings before declaring recovery.

While a change is dwelling, the page keeps showing the old status. The rationale is simple: a status page is a communication channel, not an oscilloscope. Visitors should see conditions, not noise.

No data is its own honest state#

When a check produces no value (the target is unreachable, the query returned nothing), the sample is recorded as no data rather than being guessed either way. By default that renders as its own neutral state, because Observer refuses to claim health it did not measure.

For some metrics silence genuinely means good news. An error-rate check that produces nothing usually means there were no errors. Each metric therefore has a no-data treatment: leave it as no data, or fold it to healthy, degraded, or unhealthy. The fold applies to the current status and to history, with two exceptions it never touches:

  • Days before the metric existed. The strip stays neutral before the first sample ever recorded. A new check cannot retroactively claim a month of health.
  • Hours that have not happened yet. Today's bar fills as the day progresses. Nothing colors the future.

Delayed: when the data is stale#

If a metric's agent stops reporting, the last stored status would silently freeze. Instead of asserting a stale verdict, the page marks the metric Delayed once the last sample is older than a few push intervals. Delayed metrics are excluded from the live rollup, and a group whose every metric is delayed reads Delayed rather than pretending to know. The headline says "monitoring delayed", which is the true condition: the target may be fine, Observer just cannot see it right now.

Staleness is a property of reading the page, not of the stored data. No alerts fire from staleness on the metric itself; the operator gets an agent-offline alert instead, because the broken thing is the pipeline, not the service being watched.

The 30-day bars#

Each metric shows one bar per day. A day's color is decided by the worst thing that happened in it, hour by hour: the day is divided into 24 slots, each slot carries the worst status seen in that hour, and the day summarizes the slots. Hovering a bar shows the hourly breakdown and any incidents posted that day.

Days with no samples inside the metric's lifetime count as no data (or its treatment). Days before the metric existed stay neutral. The current day only contains the hours that have already passed.

Group summary mode collapses a group to one bar per day, taking the worst status across all the group's metrics for each day.

Uptime percentages#

The uptime number next to a metric is the share of its recorded history that was healthy, over the visible window. Days without data are not counted against uptime, and also not counted for it: an unmeasured day is excluded from the calculation entirely rather than being treated as up or down.

SLOs and error budgets#

An SLO is a public promise: this metric will be healthy some percentage of the time over a rolling window. The error budget is the allowance the target leaves. A 99.9% target over 30 days allows about 43 minutes of not-healthy; the page shows how much of that allowance remains.

Budgets are computed from the same recorded samples the bars use, so the promise and the history can never tell different stories.

Two protections keep the number honest:

  • Cold start. A freshly created SLO shows "collecting" instead of a number until its metric has at least a full day of observation. With ten samples, one bad reading swings the result violently in either direction; neither the resulting panic nor the false comfort is real information.
  • Staleness. While the metric is Delayed, the budget shows no value rather than continuing to count unmeasured time for or against the promise.

Incidents and maintenance on the timeline#

Incidents affect the headline while they are open, and they annotate the history bars on the day they were posted. Scheduled maintenance shows as maintenance, distinct from an outage, while its window is running. A long-running incident stays in the headline until it is actually resolved, regardless of age. Resolved incidents remain in the history page permanently.

Why current state and history can disagree#

A page can read "All systems operational" above a strip with amber in it. That is intentional. The badge answers "can I use this service right now?" and the bars answer "how has it behaved?". A service that failed yesterday and recovered is genuinely operational today; hiding yesterday would be dishonest, and dragging yesterday into today's badge would be wrong in the other direction.

Everything on the page reduces to that principle: say exactly what was measured, when it was measured, and nothing more.

Was this page helpful?