Skip to main content
Status pushes are written to a local SQLite file before the agent attempts to deliver them to the cloud. The file is the agent’s durability layer. Pushes survive container restarts, daemon restarts, and the cloud being temporarily unreachable.

​
Behaviour

  • Every status push enqueues a row in the local SQLite file (BUFFER_PATH, default ./observer-agent-buffer.db for the binary). In a container, keep the file on a volume: set BUFFER_PATH=/data/observer-agent-buffer.db and mount a volume on /data, as the install snippets do (agent images 1.6.0 and later set this BUFFER_PATH by default). A file inside the container’s own filesystem is lost when the container is recreated, for example on upgrade.
  • A background drain controller pulls batches from the queue and posts them to the cloud’s /api/agent/receiver/batch endpoint, up to 100 samples per request (agent 1.5.0 and later). A backlog of 10,000 rows drains in about 100 requests instead of 10,000.
  • Successful posts ack and remove rows from the queue. When the cloud rejects individual rows in a batch (for example a sample with an out-of-range timestamp), those rows are dropped and logged; the rest of the batch is accepted normally.
  • Failed posts back off exponentially. The queue continues to accept new pushes during the outage.
  • When the queue reaches BUFFER_MAX_ROWS, oldest entries are evicted to admit new ones. The default is 200000 (agent 1.7.3 and later; 10000 before). Each row is one reading of roughly 300 bytes, so the default uses up to about 60 MB of disk and holds about 33 hours of readings for an agent sending 50 metrics every 30 seconds. Raise it to ride out longer outages or to cover more metrics, lower it on hosts with little disk. A full queue drains at about 100 readings per second once the cloud is reachable.
The cloud is the source of truth for historical data. The local queue is a write-ahead log that protects against transient cloud failures, not a long-term store.

​
What the operator sees

The agent dashboard’s queue panel shows three live numbers:
  • depth: rows currently waiting.
  • oldest_age_seconds: age of the oldest pending row.
  • drain_backoff_ms: current backoff between drain attempts.
A growing depth combined with a non-zero backoff is the signature of cloud unreachability. Once the cloud is reachable again, the queue drains and depth returns to near zero.

​
Queue-driven alerts

The agent reports the queue numbers on every heartbeat. A sustained high queue (depth above 1000, or an oldest pending push older than 300 seconds) raises an agent.lag_high alert that surfaces on the agent detail page and as an alert event to any destination on the Alerts page that subscribes to it. Once the queue drains, the alert clears.
Multiple replicas are fencedThe queue is a process-local file, so two agent processes sharing one AGENT_KEY would each report half the picture. The cloud resolves this by accepting data only from the most recently started process: the older process receives HTTP 409 on every push, drops those samples instead of retrying, and logs a fence warning. The dashboard flags the agent with a duplicate key warning until one process stops. If the fencing process stops heartbeating for ten minutes (for example it was killed, or a clock correction made a legitimate restart look older), the remaining process takes over automatically. Run a single replica per agent identity, or create one agent per deployment.