Skip to main content
A heartbeat check watches work that runs on a schedule: cron jobs, backups, batch exports, queue workers. Your job sends an HTTP request to a private ping URL each time it runs. Observer expects the next ping within the schedule you set, and marks the check unhealthy when it does not arrive, when the job reports a failure, or when a run goes on longer than allowed. No agent is required. Observer evaluates heartbeats itself, so a heartbeat works for jobs on any host that can make an outbound HTTPS request. Jobs without internet access can send pings through an Observer Agent instead. See Heartbeat relay.

Create a heartbeat check

1

Pick the Heartbeat source

In the console, open Metrics > New metric and choose Heartbeat as the source type. You can also start from one of the Scheduled jobs templates (nightly backup, hourly job, queue worker). See Start a metric from a template.
2

Set the schedule

Fill in the schedule fields described below, or click one of the Common schedules presets. Heartbeat metrics have no agent and no threshold fields: Observer derives the thresholds from the schedule.
3

Save and copy the ping URL

The ping URL appears on the metric’s detail page once the check is created, together with ready-made snippets for your job. The check reads No data until the first ping arrives.

Schedule fields

The schedule is an interval, not a cron expression. For a cron schedule, use its interval: a job that runs daily at 02:00 has an expected period of 1 day. Pick the grace period to cover normal variation in when the job starts and how long it runs. A nightly backup that takes up to 40 minutes might use a 1 day period with a 1 hour grace.

Ping URLs

Each check has one ping URL:
Ping URL
The URL accepts GET, POST and HEAD. Variants of it report different events: ?exit= is rejected with 400 on /start and on a URL that already carries an exit code. An exit code outside 0 to 255 is rejected rather than recorded as a success. A successful ping returns 200 with the body OK. An unknown token returns 404.

Attach output to a ping

Send a POST body with any ping to keep it with that ping, for example the tail of the job’s log. Observer keeps the first 10 KB of the body and drops the rest. The body appears in the check’s ping log and as message in the heartbeat API.

Send pings

The Send pings section on the metric detail page shows these snippets with your ping URL filled in. -fsS -m 10 --retry 3 makes curl fail on HTTP errors, stay quiet otherwise, give up after 10 seconds and retry transient failures, so the ping never blocks or breaks the job.
Shell
The GitHub Actions step goes at the end of the job’s steps. Store the ping URL as the OBSERVER_PING_URL repository secret. For the Kubernetes CronJob, store it in a Secret named observer-heartbeat under the key url.

How status is evaluated

The metric’s value is the number of seconds since the last successful run. Before the first success, it counts from the first ping. The deadline is the period plus the grace period. The thresholds stored on the metric are healthy under the deadline and unhealthy over the deadline. Both comparisons are strict, which is why a value exactly at the deadline is degraded. See Threshold operators. Observer re-evaluates every check about once a minute, so a job that stops pinging goes late within a minute of its deadline. A ping is evaluated as it arrives: a success ping recovers a late or failed check immediately. New heartbeat checks have no confirmation time (0 seconds), because the grace period already absorbs normal variation. You can add one on the metric if you want status changes held longer. See Thresholds and confirmation time.

Run duration

When a run pings /start and later finishes with a success, failure or exit code ping, Observer records the run’s duration on the finishing ping. A run that sends no /start has no duration and no max run time check.

Limits

  • 60 pings per minute per check. Further pings get 429 with a Retry-After header.
  • 10 KB of POST body kept per ping.
  • The ping log keeps the most recent 1,000 pings per check, for up to 30 days.

Keep the ping URL secret

The ping URL is the only credential needed to record pings for a check. Anyone who has it can mark the job as succeeded or failed. It cannot read anything back. Store it the way you store other secrets (a CI secret, a Kubernetes Secret, an environment variable), not in a public repository. The ping URL is shown to members of your organization and to API keys with write:metrics or write:config. It never appears on a status page. To replace a URL, open the check and click Rotate URL, or call POST /metrics/{id}/heartbeat/rotate. The old URL stops working immediately, so update your jobs right away: until they use the new URL, the check goes late.

Heartbeats and the rest of Observer

A heartbeat check is a metric. It counts toward your plan’s metric allowance, and everything that works with a metric works with it: services and SLOs, alert routing and destinations, automatic incidents and status pages. Status changes produce the same alert events as any other metric. Heartbeat metrics can also be managed as code. See Heartbeat metrics.