> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Diagnose a stalled agent

> Triage path when an agent stops reporting or its queue depth grows.

A stalled agent surfaces in two places: the cloud marks it as
**Stopped** on the Agents page, and the local dashboard's queue depth
grows beyond a few pending entries. The triage below covers both.

The agent sends a heartbeat about every 30 seconds. When heartbeats
stop:

* After **90 seconds** without one, the console shows the agent as
  **Stopped**.
* After **180 seconds**, the agent counts as offline for alerting. The
  cloud checks for offline agents every 2 minutes, so the
  `agent.offline` alert arrives 3 to 5 minutes after the last
  heartbeat. A `cleared` event follows once heartbeats resume.

## Step 1: confirm the failure mode

Open the agent's dashboard at `http://127.0.0.1:10101` on the
agent's host (it binds loopback only by default; see
[Read the agent dashboard](https://docs.use.observer/agent/guides/read-the-dashboard.md) for
containers and Kubernetes). Four patterns are common:

| Pattern | Probable cause |
| - | - |
| Process up, queue growing, last heartbeat recent but failing | Cloud reachability problem from this host. |
| Process up, queue depth zero, heartbeat succeeding, but cloud says **Stopped** | Clock skew or stale state in the console (refresh). |
| Process up, no probes have run | The cloud has not assigned any metrics to this agent yet. |
| Dashboard unreachable | The agent process is down. Container restart loop or host crash. |

## Step 2: agent down

If the dashboard is unreachable:

```bash theme={null}
# Docker
docker logs --tail 200 observer-agent

# Kubernetes
kubectl -n observer logs deploy/observer-agent --tail=200

# Linux systemd
journalctl -u observer-agent -n 200 --no-pager

# macOS launchd
tail -n 200 ~/Library/Logs/observer-agent.log
```

Look for a panic, an unhandled rejection, or a configuration error
on startup. Common causes: missing `AGENT_KEY`, malformed
`CLOUD_SERVER_URL`, port `10101` already in use.

## Step 3: cloud unreachable

If the dashboard's *Cloud link* card shows a heartbeat error
(`HB error`), the issue is between the agent and the
cloud. The published container image has no shell, so run these
checks from the agent's host, or from a debug container on the same
network. Verify in this order:

1. **DNS**: `getent hosts <cloud-host>`.

2. **TCP**: run:

   ```bash theme={null}
   curl -v https://<cloud-host>
   ```

3. **TLS**: certificate trust. For a private CA or a TLS-intercepting
   proxy, point `NODE_EXTRA_CA_CERTS` at the CA bundle.

4. **Auth**: a recently rotated `AGENT_KEY` requires updating
   the agent's environment.

The drain controller automatically retries with exponential
backoff. The queue continues to accept pushes up to its capacity
(`BUFFER_MAX_ROWS`, default `10000`). Once the cloud is reachable again,
the queue drains.

## Step 4: queue saturation

If the queue depth has hit `BUFFER_MAX_ROWS`, the oldest entries are
dropped to admit new ones. The dashboard's queue panel shows the
depth at the cap. After cloud reachability returns, the queue
drains and depth returns to near zero.

The cloud's `agent.offline` event fires when the heartbeat window
is exceeded, and is sent as an alert to any destination on the
Alerts page that subscribes to it. Subscribe to this event when
on-call needs an explicit alert.

> [!NOTE]
>
> **No probes running**
>
> The agent only runs probes for metrics the cloud has assigned to
> it. If a freshly registered agent shows no probe activity, the
> cause is on the cloud side: open the metric in the console,
> confirm its **Agent** field is set, and save.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.