> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# The local queue

> Why the agent buffers status pushes locally, and how the buffer behaves under cloud unreachability.

Status pushes are written to a local SQLite file before the agent
attempts to deliver them to the cloud. The file is the agent's
durability layer. Pushes survive container restarts, daemon
restarts, and the cloud being temporarily unreachable.

## Behaviour

* Every status push enqueues a row in the local SQLite file
  (`BUFFER_PATH`, default `./observer-agent-buffer.db` for the
  binary). In a container, keep the file on a volume: set
  `BUFFER_PATH=/data/observer-agent-buffer.db` and mount a volume on
  `/data`, as the install snippets do (agent images 1.6.0 and later set
  this `BUFFER_PATH` by default). A file inside the container's own
  filesystem is lost when the container is recreated, for example on
  upgrade.
* A background drain controller pulls batches from the queue and
  posts them to the cloud's `/api/agent/receiver/batch` endpoint,
  up to 100 samples per request (agent 1.5.0 and later). A backlog
  of 10,000 rows drains in about 100 requests instead of 10,000.
* Successful posts ack and remove rows from the queue. When the
  cloud rejects individual rows in a batch (for example a sample
  with an out-of-range timestamp), those rows are dropped and
  logged; the rest of the batch is accepted normally.
* Failed posts back off exponentially. The queue continues to
  accept new pushes during the outage.
* When the queue reaches `BUFFER_MAX_ROWS`, oldest entries are
  evicted to admit new ones. The default is `200000` (agent 1.7.3
  and later; `10000` before). Each row is one reading of roughly
  300 bytes, so the default uses up to about 60 MB of disk and holds
  about 33 hours of readings for an agent sending 50 metrics every
  30 seconds. Raise it to ride out longer outages or to cover more
  metrics, lower it on hosts with little disk. A full queue drains
  at about 100 readings per second once the cloud is reachable.

The cloud is the source of truth for historical data. The local
queue is a write-ahead log that protects against transient cloud
failures, not a long-term store.

## What the operator sees

The agent dashboard's queue panel shows three live numbers:

* `depth`: rows currently waiting.
* `oldest_age_seconds`: age of the oldest pending row.
* `drain_backoff_ms`: current backoff between drain attempts.

A growing depth combined with a non-zero backoff is the
signature of cloud unreachability. Once the cloud is reachable
again, the queue drains and depth returns to near zero.

## Queue-driven alerts

The agent reports the queue numbers on every heartbeat. A sustained
high queue (depth above 1000, or an oldest pending push older than
300 seconds) raises an `agent.lag_high` alert that surfaces on the
agent detail page and as an alert event to any destination on the
Alerts page that subscribes to it. Once the queue drains, the alert
clears.

> [!WARNING]
>
> **Multiple replicas are fenced**
>
> The queue is a process-local file, so two agent processes sharing
> one `AGENT_KEY` would each report half the picture. The cloud
> resolves this by accepting data only from the most recently
> started process: the older process receives HTTP 409 on every
> push, drops those samples instead of retrying, and logs a fence
> warning. The dashboard flags the agent with a duplicate key
> warning until one process stops. If the fencing process stops
> heartbeating for ten minutes (for example it was killed, or a
> clock correction made a legitimate restart look older), the
> remaining process takes over automatically. Run a single replica
> per agent identity, or create one agent per deployment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.