Skip to main content
A stalled agent surfaces in two places: the cloud marks it as Stopped on the Agents page, and the local dashboard’s queue depth grows beyond a few pending entries. The triage below covers both. The agent sends a heartbeat about every 30 seconds. When heartbeats stop:
  • After 90 seconds without one, the console shows the agent as Stopped.
  • After 180 seconds, the agent counts as offline for alerting. The cloud checks for offline agents every 2 minutes, so the agent.offline alert arrives 3 to 5 minutes after the last heartbeat. A cleared event follows once heartbeats resume.

Step 1: confirm the failure mode

Open the agent’s dashboard at http://127.0.0.1:10101 on the agent’s host (it binds loopback only by default; see Read the agent dashboard for containers and Kubernetes). Four patterns are common:

Step 2: agent down

If the dashboard is unreachable:
Look for a panic, an unhandled rejection, or a configuration error on startup. Common causes: missing AGENT_KEY, malformed CLOUD_SERVER_URL, port 10101 already in use.

Step 3: cloud unreachable

If the dashboard’s Cloud link card shows a heartbeat error (HB error), the issue is between the agent and the cloud. The published container image has no shell, so run these checks from the agent’s host, or from a debug container on the same network. Verify in this order:
  1. DNS: getent hosts <cloud-host>.
  2. TCP: run:
  3. TLS: certificate trust. For a private CA or a TLS-intercepting proxy, point NODE_EXTRA_CA_CERTS at the CA bundle.
  4. Auth: a recently rotated AGENT_KEY requires updating the agent’s environment.
The drain controller automatically retries with exponential backoff. The queue continues to accept pushes up to its capacity (BUFFER_MAX_ROWS, default 10000). Once the cloud is reachable again, the queue drains.

Step 4: queue saturation

If the queue depth has hit BUFFER_MAX_ROWS, the oldest entries are dropped to admit new ones. The dashboard’s queue panel shows the depth at the cap. After cloud reachability returns, the queue drains and depth returns to near zero. The cloud’s agent.offline event fires when the heartbeat window is exceeded, and is sent as an alert to any destination on the Alerts page that subscribes to it. Subscribe to this event when on-call needs an explicit alert.
No probes runningThe agent only runs probes for metrics the cloud has assigned to it. If a freshly registered agent shows no probe activity, the cause is on the cloud side: open the metric in the console, confirm its Agent field is set, and save.