> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor cron jobs with heartbeats

> Scheduled jobs ping a private URL when they run. Observer flags the check when a ping is late, a run fails, or a run takes too long.

A heartbeat check watches work that runs on a schedule: cron jobs,
backups, batch exports, queue workers. Your job sends an HTTP request
to a private ping URL each time it runs. Observer expects the next
ping within the schedule you set, and marks the check unhealthy when
it does not arrive, when the job reports a failure, or when a run
goes on longer than allowed.

No agent is required. Observer evaluates heartbeats itself, so a
heartbeat works for jobs on any host that can make an outbound HTTPS
request. Jobs without internet access can send pings through an
Observer Agent instead. See [Heartbeat relay](https://docs.use.observer/agent/guides/heartbeat-relay.md).

## Create a heartbeat check

**1. Pick the Heartbeat source**

In the console, open **Metrics** > **New metric** and choose
**Heartbeat** as the source type. You can also start from one of the
**Scheduled jobs** templates (nightly backup, hourly job, queue
worker). See [Start a metric from a template](https://docs.use.observer/docs/guides/metric-templates.md).

**2. Set the schedule**

Fill in the schedule fields described below, or click one of the
**Common schedules** presets. Heartbeat metrics have no agent and no
threshold fields: Observer derives the thresholds from the schedule.

**3. Save and copy the ping URL**

The ping URL appears on the metric's detail page once the check is
created, together with ready-made snippets for your job. The check
reads **No data** until the first ping arrives.

## Schedule fields

| Field | `source_config` key | Meaning |
| - | - | - |
| Expected every | `period_seconds` | How often the job should succeed. 60 seconds to 31 days. |
| Grace period | `grace_seconds` | Extra time after the period before the check counts as late. 0 to 7 days, default 5 minutes. |
| Max run time | `max_runtime_seconds` | Optional. With `/start` pings, a run still going after this long is unhealthy. 1 second to 7 days. Blank means no limit. |

The schedule is an interval, not a cron expression. For a cron
schedule, use its interval: a job that runs daily at 02:00 has an
expected period of 1 day.

Pick the grace period to cover normal variation in when the job
starts and how long it runs. A nightly backup that takes up to 40
minutes might use a 1 day period with a 1 hour grace.

## Ping URLs

Each check has one ping URL:

```text Ping URL theme={null}
https://use.observer/api/heartbeat/<token>
```

The URL accepts `GET`, `POST` and `HEAD`. Variants of it report
different events:

| URL | Meaning |
| - | - |
| `<ping URL>` | The run succeeded. |
| `<ping URL>/start` | A run started. Enables run duration and the max run time check. |
| `<ping URL>/fail` | The run failed. |
| `<ping URL>/<exit code>` | The run finished with this exit code (0 to 255). `0` is a success, anything else a failure. |
| `<ping URL>?exit=<code>` | Same as `/<exit code>`. Also accepted on `/fail`, which stays a failure whatever the code. |

`?exit=` is rejected with `400` on `/start` and on a URL that already
carries an exit code. An exit code outside 0 to 255 is rejected
rather than recorded as a success.

A successful ping returns `200` with the body `OK`. An unknown token
returns `404`.

### Attach output to a ping

Send a `POST` body with any ping to keep it with that ping, for
example the tail of the job's log. Observer keeps the first 10 KB of
the body and drops the rest. The body appears in the check's ping log
and as `message` in the [heartbeat API](https://docs.use.observer/api/metrics/get-metrics-by-id-heartbeat.md).

## Send pings

The **Send pings** section on the metric detail page shows these
snippets with your ping URL filled in. `-fsS -m 10 --retry 3` makes
curl fail on HTTP errors, stay quiet otherwise, give up after 10
seconds and retry transient failures, so the ping never blocks or
breaks the job.

> [tab] curl
>
> ```bash Shell theme={null}
> curl -fsS -m 10 --retry 3 -o /dev/null https://use.observer/api/heartbeat/<token>
> ```

> [tab] Cron
>
> ```bash crontab theme={null}
> # m h dom mon dow  command
> 0 2 * * *  /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://use.observer/api/heartbeat/<token>
> ```

> [tab] Start and finish
>
> ```bash Shell theme={null}
> curl -fsS -m 10 --retry 3 -o /dev/null https://use.observer/api/heartbeat/<token>/start
> if /usr/local/bin/backup.sh > /tmp/backup.log 2>&1; then
>   curl -fsS -m 10 --retry 3 -o /dev/null https://use.observer/api/heartbeat/<token>
> else
>   # Report the exit code and the last 10 KB of output.
>   code=$?
>   tail -c 10000 /tmp/backup.log | curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @- https://use.observer/api/heartbeat/<token>/$code
> fi
> ```

> [tab] GitHub Actions
>
> ```yaml .github/workflows/nightly.yml theme={null}
>       - name: Report to Observer
>         if: always()
>         env:
>           OBSERVER_PING_URL: ${{ secrets.OBSERVER_PING_URL }}
>         run: |
>           if [ "${{ job.status }}" = "success" ]; then
>             curl -fsS -m 10 --retry 3 -o /dev/null "$OBSERVER_PING_URL"
>           else
>             curl -fsS -m 10 --retry 3 -o /dev/null "$OBSERVER_PING_URL/fail"
>           fi
> ```

> [tab] Kubernetes CronJob
>
> ```yaml backup-cronjob.yaml theme={null}
> apiVersion: batch/v1
> kind: CronJob
> metadata:
>   name: backup
> spec:
>   schedule: "0 2 * * *"
>   jobTemplate:
>     spec:
>       template:
>         spec:
>           restartPolicy: OnFailure
>           containers:
>             - name: backup
>               image: alpine/curl
>               env:
>                 - name: OBSERVER_PING_URL
>                   valueFrom:
>                     secretKeyRef: { name: observer-heartbeat, key: url }
>               command: ["/bin/sh", "-c"]
>               args:
>                 - |
>                   curl -fsS -m 10 --retry 3 -o /dev/null "$OBSERVER_PING_URL/start"
>                   /run-job.sh && curl -fsS -m 10 --retry 3 -o /dev/null "$OBSERVER_PING_URL" \
>                     || curl -fsS -m 10 --retry 3 -o /dev/null "$OBSERVER_PING_URL/fail"
> ```

> [tab] Via agent
>
> For a job that cannot reach the internet but can reach an Observer
> agent with the [heartbeat relay](https://docs.use.observer/agent/guides/heartbeat-relay.md)
> enabled. The paths are the same as on the ping URL, with the agent's
> address in front. Replace `<agent-host>` with the agent's address.
>
> ```bash Shell theme={null}
> curl -fsS -m 10 --retry 3 -o /dev/null http://<agent-host>:10102/heartbeat/<token>/start
> /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null http://<agent-host>:10102/heartbeat/<token> \
>   || curl -fsS -m 10 --retry 3 -o /dev/null http://<agent-host>:10102/heartbeat/<token>/fail
> ```
>
> ```bash Agent environment theme={null}
> HEARTBEAT_RELAY_ENABLED=true
> # Listen beyond loopback so other hosts or pods can reach it:
> HEARTBEAT_RELAY_HOST=0.0.0.0
> HEARTBEAT_RELAY_PORT=10102
> ```
>
> Pings sent this way appear in the ping log marked as sent via agent.
> The relay requires agent 1.6.0 or later.

The GitHub Actions step goes at the end of the job's `steps`. Store
the ping URL as the `OBSERVER_PING_URL` repository secret. For the
Kubernetes CronJob, store it in a Secret named `observer-heartbeat`
under the key `url`.

## How status is evaluated

The metric's value is the number of seconds since the last successful
run. Before the first success, it counts from the first ping. The
deadline is the period plus the grace period.

| Situation | Status |
| - | - |
| No ping received yet | No data |
| Seconds since last success is under the deadline | healthy |
| Seconds since last success is exactly the deadline | degraded |
| Seconds since last success is over the deadline | unhealthy (late) |
| The latest finished run reported a failure | unhealthy until the next success ping |
| A run that pinged `/start` has been going longer than the max run time | unhealthy until it finishes |

The thresholds stored on the metric are healthy `under` the deadline
and unhealthy `over` the deadline. Both comparisons are strict, which
is why a value exactly at the deadline is degraded. See
[Threshold operators](https://docs.use.observer/docs/reference/threshold-operators.md).

Observer re-evaluates every check about once a minute, so a job that
stops pinging goes late within a minute of its deadline. A ping is
evaluated as it arrives: a success ping recovers a late or failed
check immediately.

New heartbeat checks have no confirmation time (0 seconds), because the grace period
already absorbs normal variation. You can add one on the metric if
you want status changes held longer. See
[Thresholds and confirmation time](https://docs.use.observer/docs/concepts/thresholds-and-dwell.md).

### Run duration

When a run pings `/start` and later finishes with a success, failure
or exit code ping, Observer records the run's duration on the
finishing ping. A run that sends no `/start` has no duration and no
max run time check.

## Limits

* 60 pings per minute per check. Further pings get `429` with a
  `Retry-After` header.
* 10 KB of `POST` body kept per ping.
* The ping log keeps the most recent 1,000 pings per check, for up to
  30 days.

## Keep the ping URL secret

The ping URL is the only credential needed to record pings for a
check. Anyone who has it can mark the job as succeeded or failed. It
cannot read anything back. Store it the way you store other secrets
(a CI secret, a Kubernetes Secret, an environment variable), not in a
public repository.

The ping URL is shown to members of your organization and to API keys
with `write:metrics` or `write:config`. It never appears on a status
page.

To replace a URL, open the check and click **Rotate URL**, or call
[`POST /metrics/{id}/heartbeat/rotate`](https://docs.use.observer/api/metrics/post-metrics-by-id-heartbeat-rotate).
The old URL stops working immediately, so update your jobs right
away: until they use the new URL, the check goes late.

## Heartbeats and the rest of Observer

A heartbeat check is a metric. It counts toward your plan's metric
allowance, and everything that works with a metric works with it:
services and SLOs, alert routing and destinations, automatic
incidents and status pages. Status changes produce the same alert
events as any other metric.

Heartbeat metrics can also be managed as code. See
[Heartbeat metrics](https://docs.use.observer/cli/overview.md#heartbeat-metrics).

## Related

* [Heartbeat relay](https://docs.use.observer/agent/guides/heartbeat-relay.md)
* [Heartbeat API](https://docs.use.observer/api/metrics/get-metrics-by-id-heartbeat.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.