Observer
Observer Agent

The local queue

Why the agent buffers status pushes locally, and how the buffer behaves under cloud unreachability.

Status pushes are written to a local SQLite file before the agent attempts to deliver them to the cloud. The file is the agent's durability layer. Pushes survive container restarts, daemon restarts, and the cloud being temporarily unreachable.

Behaviour

  • Every status push enqueues a row in the local SQLite file (BUFFER_PATH, default ./observer-agent-buffer.db).
  • A background drain controller pulls batches from the queue and posts them to the cloud's /api/agent/receiver endpoint.
  • Successful posts ack and remove rows from the queue.
  • Failed posts back off exponentially. The queue continues to accept new pushes during the outage.
  • When the queue reaches BUFFER_MAX_ROWS (default 10000), oldest entries are evicted to admit new ones.

The cloud is the source of truth for historical data. The local queue is a write-ahead log that protects against transient cloud failures, not a long-term store.

What the operator sees

The agent dashboard's queue panel shows three live numbers:

  • depth: rows currently waiting.
  • oldest_age_seconds: age of the oldest pending row.
  • drain_backoff_ms: current backoff between drain attempts.

A growing depth combined with a non-zero backoff is the signature of cloud unreachability. Once the cloud is reachable again, the queue drains and depth returns to near zero.

Cloud-side signals

The cloud's heartbeat receiver computes two derived signals from the queue numbers in every heartbeat:

  • agent.lag_high: opens when queue_depth > 1000 or queue_oldest_age_seconds > 300. Surfaces in the agent detail page and as a webhook event when subscribed.
  • agent.uptime_degraded: opens when 24-hour uptime falls below 95%.

Both signals clear with 60-second hysteresis to avoid flapping.

Was this page helpful?