Run behind a proxy or in a restricted network

This page covers installing the agent in networks that restrict outbound traffic: an HTTP proxy is mandatory, a TLS-intercepting proxy re-signs traffic with a corporate CA, internal services use a private CA, or the host cannot pull from public registries.

Outbound requirements

The agent only makes outbound connections. Nothing needs to reach the agent from the internet, and no inbound firewall rule is required.

Destination	Port	When	Purpose
Observer Cloud (`use.observer`, or the host in your `CLOUD_SERVER_URL`)	443 (HTTPS)	Always	Heartbeats, metric definitions, result pushes, optional log broadcast. Every request goes to a path under `/api/agent/` and carries the `Agent-Key` header.
Your probe targets	Whatever each probe uses	Always	Prometheus, HTTP endpoints, databases, DNS servers and the rest. These are usually internal.
`ghcr.io`	443	Install and upgrade only	Pulling the container image. Layer downloads are redirected to `pkg-containers.githubusercontent.com`.
`github.com`	443	Install and upgrade only	Downloading a release binary. Downloads are redirected to a GitHub asset host (currently `release-assets.githubusercontent.com`).

The agent does not follow redirects from Observer Cloud, so a proxy or gateway that answers with a 3xx (for example a captive portal or a login page) shows up as a failed request in the agent logs, not as a silent success. Everything the agent sends to the cloud is described in The agent / cloud boundary.

Use an HTTP proxy

The agent’s runtime honours the standard proxy variables. Set them in the agent’s environment:

Variable	Example	Notes
`HTTPS_PROXY`	`http://proxy.corp.example:3128`	Proxy for HTTPS requests, including the connection to Observer Cloud. The URL scheme is the scheme used to talk to the proxy, which is usually `http://` even for HTTPS traffic.
`HTTP_PROXY`	`http://proxy.corp.example:3128`	Proxy for plain HTTP requests.
`NO_PROXY`	`localhost,127.0.0.1,prometheus.internal,.corp.example`	Comma-separated hosts and domain suffixes that bypass the proxy. A leading dot matches every subdomain. CIDR ranges are not supported; list hostnames or individual IP addresses.

If the proxy needs credentials, put them in the URL: http://user:[email protected]:3128. Treat the value as a secret.

Keep internal probe targets out of the proxy

The proxy variables apply to every HTTP request the agent makes, not just the connection to Observer Cloud. That includes Prometheus queries and HTTP probes. Most probe targets are internal services that the proxy either cannot reach or should not see, so add them to NO_PROXY:

The Prometheus host from PROMETHEUS_SERVER_URL.
The hosts of internal HTTP, Loki and Elasticsearch targets.
localhost and 127.0.0.1.
In Kubernetes, the cluster domain suffix (for example .svc,.svc.cluster.local), so in-cluster Services are reached directly.

Probes that open their own connections (TCP, DNS, ICMP, TLS certificate and database probes) connect directly to the target and are not routed through the proxy. gRPC and WebSocket probes use their own client libraries; confirm their behaviour with a test probe before relying on the proxy for them.

TLS-intercepting proxies and private CAs

A TLS-intercepting proxy (also called TLS inspection or SSL inspection) terminates HTTPS and re-signs it with a corporate CA. The agent rejects that certificate unless it trusts the CA, and logs a certificate verification error on every request to the cloud. Add the CA to the agent’s trust store with NODE_EXTRA_CA_CERTS: point it at a PEM file on the agent host (or inside the container) that holds the CA certificate, or several certificates concatenated. The runtime trusts these certificates in addition to its built-in trust store, so public endpoints keep working. The same variable covers internal HTTPS targets signed by a private CA: Prometheus, Loki, Elasticsearch and HTTP probes all trust the extra certificates. NODE_EXTRA_CA_CERTS is read when the process starts. After you change the file or the variable, restart the agent. Ask your network or security team for the CA certificate in PEM format (a text file that starts with -----BEGIN CERTIFICATE-----). If you only have a DER file (.cer or .crt with binary content), convert it:

openssl x509 -inform der -in corp-ca.cer -out corp-ca.pem

The container image runs as a non-root user (UID 65532), so make sure a mounted PEM file is world-readable (chmod 644).

Per-probe CA certificates

Some probes take a CA certificate in their own configuration instead. This scopes the trust to that one probe:

gRPC probes: set ca_cert_ref in TLS or mTLS mode. See gRPC probes.
HTTP probes: ca_cert_ref applies together with mTLS client certificates. See mTLS authentication. For a plain HTTPS target behind a private CA, use NODE_EXTRA_CA_CERTS.

ca_cert_ref holds the name of an environment variable on the agent host, never the certificate itself. The variable’s value is either the PEM text or a path to a PEM file.

Do not disable TLS verification

Two variables turn certificate checks off. Neither belongs in production:

SKIP_SSL_VERIFICATION=true disables verification on the connection to Observer Cloud. Anyone who can intercept that connection can read your agent key and feed the agent false metric definitions.
NODE_TLS_REJECT_UNAUTHORIZED=0 disables verification for every HTTPS request the agent makes: the cloud, Prometheus, and every probe. An HTTP probe then reports a target as healthy even when its certificate is invalid or forged.

If verification fails, fix the trust store with NODE_EXTRA_CA_CERTS instead. To accept a self-signed certificate on a single internal HTTP probe, set verify_tls: false on that probe only.

Examples

Each example sets a proxy, bypasses it for internal targets, and trusts a corporate CA stored in corp-ca.pem. Remove the parts you do not need.

Docker
Compose
Kubernetes
systemd

docker run -d \
  --name observer-agent \
  --restart unless-stopped \
  -v observer-agent-buffer:/data \
  -v /etc/observer/corp-ca.pem:/etc/observer/corp-ca.pem:ro \
  -e AGENT_KEY=obs_live_... \
  -e CLOUD_SERVER_URL=https://use.observer \
  -e BUFFER_PATH=/data/observer-agent-buffer.db \
  -e HTTPS_PROXY=http://proxy.corp.example:3128 \
  -e HTTP_PROXY=http://proxy.corp.example:3128 \
  -e NO_PROXY=localhost,127.0.0.1,prometheus.internal,.corp.example \
  -e NODE_EXTRA_CA_CERTS=/etc/observer/corp-ca.pem \
  ghcr.io/useobserver/agent:1.6.0

docker-compose.yml

services:
  observer-agent:
    image: ghcr.io/useobserver/agent:1.6.0
    container_name: observer-agent
    restart: unless-stopped
    env_file: [.env]
    environment:
      BUFFER_PATH: /data/observer-agent-buffer.db
      HTTPS_PROXY: http://proxy.corp.example:3128
      HTTP_PROXY: http://proxy.corp.example:3128
      NO_PROXY: localhost,127.0.0.1,prometheus.internal,.corp.example
      NODE_EXTRA_CA_CERTS: /etc/observer/corp-ca.pem
    volumes:
      - observer-agent-buffer:/data
      - ./corp-ca.pem:/etc/observer/corp-ca.pem:ro

volumes:
  observer-agent-buffer:

Keep AGENT_KEY (and proxy credentials, if any) in .env.

Store the CA in a ConfigMap (a CA certificate is public, so a Secret is optional):

kubectl -n observer create configmap observer-agent-ca \
  --from-file=corp-ca.pem=./corp-ca.pem

Then add the variables and the volume to the Deployment from Install on Kubernetes:

agent.yaml (excerpt)

    spec:
      containers:
        - name: agent
          image: ghcr.io/useobserver/agent:1.6.0
          env:
            - name: AGENT_KEY
              valueFrom:
                secretKeyRef: { name: observer-agent, key: agent-key }
            - name: CLOUD_SERVER_URL
              value: https://use.observer
            - name: HTTPS_PROXY
              value: http://proxy.corp.example:3128
            - name: HTTP_PROXY
              value: http://proxy.corp.example:3128
            - name: NO_PROXY
              value: localhost,127.0.0.1,.svc,.svc.cluster.local,.corp.example
            - name: NODE_EXTRA_CA_CERTS
              value: /etc/observer/ca/corp-ca.pem
          volumeMounts:
            - { name: data, mountPath: /data }
            - { name: corp-ca, mountPath: /etc/observer/ca, readOnly: true }
      volumes:
        - name: data
          emptyDir: {}
        - name: corp-ca
          configMap:
            name: observer-agent-ca

If the proxy URL carries credentials, put HTTPS_PROXY in the observer-agent Secret and reference it with secretKeyRef, like AGENT_KEY. To use a Secret for the CA instead of a ConfigMap, replace the configMap volume with secret: { secretName: observer-agent-ca }.

Add the variables to the environment file used by the unit from Install from a release binary:

/etc/observer-agent.env

AGENT_KEY=obs_live_...
CLOUD_SERVER_URL=https://use.observer
HTTPS_PROXY=http://proxy.corp.example:3128
HTTP_PROXY=http://proxy.corp.example:3128
NO_PROXY=localhost,127.0.0.1,prometheus.internal,.corp.example
NODE_EXTRA_CA_CERTS=/etc/observer/corp-ca.pem

Or set them directly in the unit:

/etc/systemd/system/observer-agent.service (excerpt)

[Service]
EnvironmentFile=/etc/observer-agent.env
Environment=HTTPS_PROXY=http://proxy.corp.example:3128
Environment=NO_PROXY=localhost,127.0.0.1,prometheus.internal,.corp.example
Environment=NODE_EXTRA_CA_CERTS=/etc/observer/corp-ca.pem
ExecStart=/usr/local/bin/observer-agent

sudo install -m 644 corp-ca.pem /etc/observer/corp-ca.pem
sudo systemctl daemon-reload
sudo systemctl restart observer-agent

The binary reads NODE_EXTRA_CA_CERTS from the file system of the host, so the path must be readable by the service user.

Install without access to public registries

The host that runs the agent does not need to reach ghcr.io or github.com after installation. Fetch the artifact on a machine that can, then move it across.

Container image: save and load

On a machine with internet access, pull the image for the architecture of the target host and save it to a file:

docker pull --platform linux/amd64 ghcr.io/useobserver/agent:1.6.0
docker save ghcr.io/useobserver/agent:1.6.0 -o observer-agent-1.6.0.tar
sha256sum observer-agent-1.6.0.tar

Use --platform linux/arm64 for ARM hosts. Copy the tar file to the target host, check the checksum, and load it:

sha256sum observer-agent-1.6.0.tar
docker load -i observer-agent-1.6.0.tar

Then run the agent with the same image reference.

Container image: mirror into an internal registry

If your organisation runs its own registry, copy the image into it and point the install at the mirror. ghcr.io/useobserver/agent is a multi-architecture image (linux/amd64 and linux/arm64); use a tool that copies every architecture, such as skopeo or crane:

skopeo copy --all \
  docker://ghcr.io/useobserver/agent:1.6.0 \
  docker://registry.corp.example/observer/agent:1.6.0

# or
crane copy ghcr.io/useobserver/agent:1.6.0 registry.corp.example/observer/agent:1.6.0

Replace ghcr.io/useobserver/agent with registry.corp.example/observer/agent in the Docker, Compose or Kubernetes examples. Pin exact versions so every host runs the image you mirrored.

Release binary: download and transfer

On a machine with internet access, download the binary and the checksum file from github.com/useobserver/agent/releases:

VERSION=1.6.0
curl -fLO https://github.com/useobserver/agent/releases/download/agent-v${VERSION}/observer-agent-linux-x64
curl -fLO https://github.com/useobserver/agent/releases/download/agent-v${VERSION}/SHA256SUMS.txt

Copy both files to the target host and verify the binary there, after the transfer:

sha256sum -c SHA256SUMS.txt --ignore-missing
chmod +x observer-agent-linux-x64
sudo mv observer-agent-linux-x64 /usr/local/bin/observer-agent

Then continue with the systemd steps in Install from a release binary.

Air-gapped and isolated networks

Observer Cloud is a hosted service, so the agent needs an outbound HTTPS path to it, directly or through a proxy. No inbound connection is ever required. A network with no route to Observer Cloud at all is not supported: the agent can run and collect readings there, but nothing reaches your status pages. What the agent does tolerate is losing that path for a while. Every result is written to the local queue first, a SQLite file at BUFFER_PATH, and a background process delivers it to the cloud in batches. When the cloud is unreachable:

The agent keeps probing and keeps queueing results.
Delivery retries with exponential backoff, from 1 second up to a maximum of 5 minutes between attempts.
The queue holds up to BUFFER_MAX_ROWS rows (default 10000). Once it is full, the oldest rows are evicted to make room and the agent logs an error with the number dropped.
So that one bad row cannot block the queue forever, a row that fails 20 delivery attempts in a row is dropped with an error log. During a long outage this costs at most one row roughly every hour or two.
When the path returns, the backlog drains in batches of up to 100 rows per request, oldest first.

How long an outage the default cap covers depends on how many probes the agent runs and how often. Raise BUFFER_MAX_ROWS if you expect long interruptions, and keep BUFFER_PATH on a persistent volume so the queue survives a restart. While the agent cannot reach the cloud it also stops sending heartbeats, so the console marks it offline and the agent.offline alert fires even though probing continues locally. For jobs on hosts that cannot reach the internet at all, the heartbeat relay lets them report heartbeat checks through the agent instead.

Verify connectivity

After starting the agent, check three places. Agent logs. A working connection logs Successfully fetched metric definitions within about 30 seconds. A failed one logs Error fetching metric definitions: followed by the cause. Heartbeat failures are only logged with VERBOSE=true, which is worth setting while you debug the network path. Typical causes:

Log content	Likely cause
`unable to get local issuer certificate`, `self signed certificate in certificate chain`	A TLS-intercepting proxy or private CA that the agent does not trust. Set `NODE_EXTRA_CA_CERTS`.
`ECONNREFUSED`, `ETIMEDOUT`, `DNS`	No route to the cloud. Set `HTTPS_PROXY`, or ask for an egress rule to port 443.
`407` or a proxy authentication error	The proxy requires credentials. Add them to the proxy URL.
`redirect (not followed)`	Something between the agent and the cloud answers with a redirect, such as a captive portal.
`HTTP 401`	The agent key is wrong or was rotated.

Where to read the logs: docker logs -f observer-agent, docker compose logs -f observer-agent, kubectl -n observer logs -f deploy/observer-agent, or journalctl -u observer-agent -f. Local dashboard. The agent serves a read-only dashboard on port 10101, bound to loopback by default. Its Cloud link card shows the last heartbeat and the last push, with the error text when one failed, and the queue card shows whether results are piling up. See Read the agent dashboard. Console. The agent’s page in the console shows when it last checked in. A connected agent is marked running within 90 seconds of starting. If it stays offline while the logs show no errors, see Diagnose a stalled agent. To test the network path from the agent host before installing, make a request through the same proxy. Pass --cacert only when a TLS-intercepting proxy re-signs the traffic, because it replaces curl’s default trust store:

curl -sS -o /dev/null -w '%{http_code}\n' \
  --proxy http://proxy.corp.example:3128 \
  --cacert /etc/observer/corp-ca.pem \
  https://use.observer/

Any HTTP status code means the path and the certificate chain work. A TLS error means the CA file is wrong or incomplete.

Quickstart

Concepts

Guides

Reference

Run behind a proxy or in a restricted network

Outbound requirements

Use an HTTP proxy

Keep internal probe targets out of the proxy

TLS-intercepting proxies and private CAs

Per-probe CA certificates

Do not disable TLS verification

Examples

Install without access to public registries

Container image: save and load

Container image: mirror into an internal registry

Release binary: download and transfer

Air-gapped and isolated networks

Verify connectivity

​Outbound requirements

​Use an HTTP proxy

​Keep internal probe targets out of the proxy

​TLS-intercepting proxies and private CAs

​Per-probe CA certificates

​Do not disable TLS verification

​Examples

​Install without access to public registries

​Container image: save and load

​Container image: mirror into an internal registry

​Release binary: download and transfer

​Air-gapped and isolated networks

​Verify connectivity

Outbound requirements

Use an HTTP proxy

Keep internal probe targets out of the proxy

TLS-intercepting proxies and private CAs

Per-probe CA certificates

Do not disable TLS verification

Examples

Install without access to public registries

Container image: save and load

Container image: mirror into an internal registry

Release binary: download and transfer

Air-gapped and isolated networks

Verify connectivity