Observer Docs

The agent emits a heartbeat to the cloud on a fixed interval (typically every 30 seconds). The payload is the agent’s view of its own runtime state. The cloud uses the payload to compute 24-hour uptime, restart counts, and lag alerts.

Endpoint

POST <CLOUD_SERVER_URL>/api/agent/heartbeat
Authorization: provided via Agent-Key header (key transport detail)
Content-Type: application/json

Body

{
  "version": "1.2.3",
  "uptime_seconds": 12345,
  "buffer_size": 0,
  "buffer_oldest_age_seconds": 0,
  "queue_depth": 0,
  "queue_oldest_age_seconds": 0,
  "queue_capacity": 10000,
  "agent_started_at": "2026-05-09T12:00:00Z",
  "source_types_active": ["prometheus", "http", "tcp"],
  "custom_probes": [
    { "name": "queue_lag", "has_config_schema": false }
  ],
  "build": {
    "channel": "official",
    "commit": "0a1b2c3d4e5f",
    "source_hash": "9f2c4a..."
  },
  "hostname": "ip-10-0-3-17"
}

Field reference

Field	Type	Meaning
`version`	string	Build-time version of the agent.
`uptime_seconds`	integer	Wall-clock seconds since process start.
`buffer_size`	integer	Deprecated alias for `queue_depth`; kept for older cloud versions.
`buffer_oldest_age_seconds`	integer	Legacy alias for `queue_oldest_age_seconds`.
`queue_depth`	integer	Pushes currently waiting in the local queue.
`queue_oldest_age_seconds`	integer	Age of the oldest pending push, in seconds.
`queue_capacity`	integer	Hard cap on the queue (`BUFFER_MAX_ROWS`).
`agent_started_at`	ISO timestamp	When the process started. The cloud uses changes to this value to detect restarts.
`source_types_active`	string[]	Distinct source types of the metric definitions currently assigned to the agent.
`otlp_stats`	object	OTLP receiver counters, sent when the agent runs the receiver: `running`, `listen_addr`, `bearer_required`, `data_points_received`, `data_points_dropped`, `requests_authenticated`, `requests_rejected_auth`, `requests_rejected_payload`, `unique_streams`, `active_subscriptions`. Omitted otherwise.
`custom_probes`	object[]	Custom probes registered on the agent, each with `name` and `has_config_schema`. The probe function and its `description` are never sent (agents before 1.7.0 also sent `description`; Observer discards it). Sent as an empty list when none are registered, so removing a probe clears it in the console; older agents omit the field. The console fills its custom probe list from this field.
`build`	object	Build provenance (agent 1.5.0 and later). `channel` is `official` when the process runs the published image, `source` when it runs from a checkout. `source_hash` is a deterministic hash of the agent’s source tree; `commit` is the git commit the official image was built from. See Build provenance.
`hostname`	string	The agent host’s name, as the operating system reports it. Shown on the agent detail page. Older agents omit it.

Response

The cloud acknowledges a heartbeat with a small JSON body:

{
  "ok": true,
  "uptime_pct_24h": 99.9,
  "restart_count_24h": 0,
  "definitions_version": "5f0c2a9e41b7d3c8a6e1f2b4c9d0e7a3"
}

definitions_version is an opaque string that changes whenever the metric definitions assigned to this agent change: a metric is created, edited, deleted, or moved to or from this agent. The GET /api/agent/metrics-definitions response carries the same value in its X-Observer-Definitions-Version header. When a heartbeat returns a version that differs from the one of the agent’s last fetch, the agent fetches its definitions again right away, so a change in the console reaches the agent within about one heartbeat interval (30 seconds). Agent 1.6.0 and later act on it. When two processes share one agent key, the response to the process that lost ownership is { "ok": true, "fenced": true } with no other fields. See Duplicate key detection. The definitions_version field is optional. An agent that does not recognise it, or a cloud that does not send it, falls back to the periodic fetch: every 30 seconds for the first 10 minutes after start while the agent has no definitions, then every 5 minutes. Agents older than 1.6.0 fetch every 5 minutes.

Lag and uptime alerts

The queue and uptime numbers in the heartbeat drive two alerts:

agent.lag_high: opens when queue_depth is above 1000 or queue_oldest_age_seconds is above 300.
agent.uptime_degraded: opens when uptime over the last 24 hours falls below 95%.

These alerts surface on the agent detail page and as alert events to any destination on the Alerts page that subscribes to them. A separate agent.offline event fires when the cloud stops receiving heartbeats from the agent.

Duplicate key detection

The agent sends its start time in an Agent-Started-At header on every request. When two processes share one agent key, the cloud treats the most recently started process as the owner:

Heartbeats from the older process are acknowledged but ignored. They do not count toward uptime and do not update the dashboard.
Pushes from the older process are rejected with HTTP 409. The agent drops the rejected samples instead of retrying and logs a warning at most once every five minutes.
The dashboard shows a duplicate key warning on the agent, and an agent.duplicate_key alert event fires (state open) to any destination on the Alerts page that subscribes to it. Once the older process goes quiet for ten minutes, the warning clears and a cleared event follows.

Restart detection is monotonic: only a strictly newer agent_started_at records a restart. There is one escape hatch: when the process holding the newer start time stops heartbeating for ten minutes, a fenced process may take over as the owner. This recovers a single agent whose clock moved backwards across a restart without operator action. To resolve a duplicate key situation, stop one of the two processes, or rotate the key and give each deployment its own agent.

Versioning

Field additions to the payload are additive. Older agents that omit newer fields stay compatible; older cloud versions ignore fields they do not recognise.

Quickstart

Concepts

Guides

Reference

Heartbeat payload

Endpoint

Body

Field reference

Response

Lag and uptime alerts

Duplicate key detection

Versioning

​Endpoint

​Body

​Field reference

​Response

​Lag and uptime alerts

​Duplicate key detection

​Versioning

Endpoint

Body

Field reference

Response

Lag and uptime alerts

Duplicate key detection

Versioning