The local queue
Why the agent buffers status pushes locally, and how the buffer behaves under cloud unreachability.
Status pushes are written to a local SQLite file before the agent attempts to deliver them to the cloud. The file is the agent's durability layer. Pushes survive container restarts, daemon restarts, and the cloud being temporarily unreachable.
Behaviour
- Every status push enqueues a row in the local SQLite file
(
BUFFER_PATH, default./observer-agent-buffer.db). - A background drain controller pulls batches from the queue and
posts them to the cloud's
/api/agent/receiverendpoint. - Successful posts ack and remove rows from the queue.
- Failed posts back off exponentially. The queue continues to accept new pushes during the outage.
- When the queue reaches
BUFFER_MAX_ROWS(default10000), oldest entries are evicted to admit new ones.
The cloud is the source of truth for historical data. The local queue is a write-ahead log that protects against transient cloud failures, not a long-term store.
What the operator sees
The agent dashboard's queue panel shows three live numbers:
depth: rows currently waiting.oldest_age_seconds: age of the oldest pending row.drain_backoff_ms: current backoff between drain attempts.
A growing depth combined with a non-zero backoff is the signature of cloud unreachability. Once the cloud is reachable again, the queue drains and depth returns to near zero.
Cloud-side signals
The cloud's heartbeat receiver computes two derived signals from the queue numbers in every heartbeat:
agent.lag_high: opens whenqueue_depth > 1000orqueue_oldest_age_seconds > 300. Surfaces in the agent detail page and as a webhook event when subscribed.agent.uptime_degraded: opens when 24-hour uptime falls below 95%.
Both signals clear with 60-second hysteresis to avoid flapping.