Observer Docs

HTTP probes hit a URL on the configured interval. The reported value is response_time_ms for successful requests, or no_data with a reason code when the request fails (timeout, connection refused, body mismatch, unexpected status).

Configuration shape

{
  "url": "https://api.example.com/healthz",
  "method": "GET",
  "expected_status": 200,
  "timeout_ms": 5000,
  "headers": { "User-Agent": "observer-agent" },
  "header_refs": { "Authorization": "OBSERVER_API_TOKEN" },
  "body_match": "ok",
  "follow_redirects": true,
  "verify_tls": true
}

Field reference

Field	Default	Notes
`url`	required	Full http(s) URL. URLs with embedded credentials (`user:pass@`) are rejected; use `header_refs`.
`method`	`GET`	One of `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`.
`expected_status`	`200`	Single integer or array. The probe matches if the response code is in the set.
`timeout_ms`	`5000`	Aborts the request when exceeded. Reports `ETIMEDOUT`.
`headers`	none	Extra non-secret request headers, stored as entered. Names that look like credentials, matched case-insensitively (`Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`, `X-API-Key`, `Api-Key`, `ApiKey`, anything containing `token`, `secret`, `password`, `passwd`, `auth`, or `session`, or ending in `key`), are rejected here; use `header_refs`.
`header_refs`	none	Secret request headers. Maps a header name to the name of an env var on the agent host; the agent sends that env var’s value. Agent 1.6.0 and later. See Secret headers.
`body_match`	none	Optional substring match against the first 4KB of the response body. Mismatch reports `body_mismatch`.
`follow_redirects`	`true`	When `false`, redirect responses count against `expected_status`.
`verify_tls`	`true`	When `false`, the probe accepts invalid TLS certificates. Useful for self-signed internal endpoints.
`json_path`	none	Optional JSONPath expression. When set, the probe value is the number extracted from the JSON response at this path, not the response time. See JSON path extraction.
`client_cert_ref`	none	Env-var name on the agent holding the client certificate PEM (or a path to it). Enables mTLS. See mTLS authentication.
`client_key_ref`	none	Env-var name holding the client private key PEM (or a path to it). Required with `client_cert_ref`. See mTLS authentication.
`ca_cert_ref`	none	Optional env-var name holding a CA cert PEM (or a path to one) to verify the server. When set, the probe trusts only that CA; when unset, the system trust store. Works with or without mTLS from agent 1.6.1; earlier versions only read it together with a client certificate.

Reason codes

The full catalog, with the hint the console shows for each code, is in Reason codes. The reason field on no_data results is a stable code. The agent normalises network errors from its runtime into one set, shared with the TCP, DNS, TLS certificate and Prometheus sources:

ETIMEDOUT: request exceeded timeout_ms.
ECONNREFUSED: the host resolved but refused the connection (port closed, service down, or a firewall rejecting it).
ENOTFOUND, EAI_AGAIN: DNS resolution failed.
ECONNRESET: the target closed the connection before answering.
EHOSTUNREACH: no network route from the agent to the target.
connect_failed: the connection could not be opened for another reason.
CERT_HAS_EXPIRED, DEPTH_ZERO_SELF_SIGNED_CERT, ERR_TLS_CERT_ALTNAME_INVALID and the other certificate codes, or tls_handshake_failed: the TLS handshake failed.
ConnectionRefused: reported by older agents for both a refused port and a hostname that does not resolve. The console shows it as Could not connect. Current agents report ECONNREFUSED or ENOTFOUND instead.
invalid_url: url could not be parsed.
invalid_url_scheme: url uses a scheme other than http or https.
http_error: the request failed without a more specific error code.
unexpected_status:<code>: status code not in expected_status.
body_mismatch: body_match was set and the response body did not contain it.
header_ref_missing: an env var named in header_refs is not set (or is blank) on the agent host. The request is not sent.
header_ref_invalid: an env var named in header_refs holds a line break or control character, which is not allowed in a header.

Body match limitOnly the first 4KB of the response body is read. If the marker string is later in the response, the probe reports body_mismatch. Move the marker earlier in the response, or use a dedicated health endpoint that returns it in the first kilobyte.

Threshold examples

Goal	Healthy	Unhealthy
Endpoint reachable, fast	`under 500`	`over 2000`
Endpoint reachable	`under 5000`	`over 10000`

For pure reachability with no latency requirement, set the unhealthy threshold equal to the timeout and rely on no_data for failures.

JSON path extraction

Set json_path when the value worth thresholding lives inside the response body, not in the response time. The agent fetches the URL, parses the body as JSON, applies the JSONPath expression, and uses the extracted value as the probe value.

{
  "url": "https://api.example.com/internal/queues",
  "method": "GET",
  "expected_status": 200,
  "json_path": "$.pending_count"
}

If pending_count is 42 in the response, the probe value is 42. Threshold on it like any other numeric probe.

When to use it

A custom health endpoint that returns a payload like {"status": "ok", "queue_depth": 42} and you want to alert on queue depth, not just status code.
A third-party JSON API where the field you care about is buried in the response.
An internal metric endpoint that returns several numbers; pick one per probe.

When not to use it

The endpoint exposes Prometheus exposition format. Use the prometheus source type instead, which understands the format natively and handles labels.
The value lives across multiple fields and needs computation (sum, ratio, percentile). Compute it at the source and expose a single field, or move to a Prometheus-style metric.
The response is large (multi-MB). The agent caps the body at 10 MB and reports json_body_too_large. Add a dedicated probe endpoint instead.

Path syntax

The agent evaluates paths with the jsonpath-plus dialect of JSONPath. Common patterns:

Path	Matches
`$.field`	Top-level field.
`$.parent.child`	Nested field.
`$.items[0]`	First element of an array.
`$.items[-1:]`	Last element of an array.
`$.items[?(@.name=='checkout')].count`	Filtered array element.

Value rules

The expression must resolve to exactly one value. The agent coerces:

Numbers are used directly.
Booleans: true becomes 1, false becomes 0.
Strings that parse as finite numbers (e.g. "42", "3.14") become the parsed number. Surrounding whitespace is ignored.
Anything else reports no_data with a specific reason code (see below).

Reason codes specific to JSON path

json_parse_failed: response body was not valid JSON.
json_path_invalid: the json_path expression could not be evaluated. Check its syntax.
json_path_no_match: the path matched nothing in the response.
json_path_multi_match: the path matched more than one value. Tighten the expression.
json_path_non_numeric: the matched value was a string that could not be parsed as a number, or another non-numeric scalar.
json_path_unsupported_type: the matched value was an array, object, or null. The path must point at a leaf value.
json_body_too_large: the response exceeded 10 MB. The agent caps body reads to protect memory.

Body match still appliesbody_match and json_path work together. If both are set, the body must contain the substring AND the JSONPath must extract a numeric value. Use body_match as a coarse sanity check (e.g. the response includes the word ok) and json_path for the actual probe value.

Troubleshooting

The probe always returns json_parse_failed. Check the endpoint’s Content-Type. The agent does not require application/json, but the body must be parseable as JSON. If the response is text/plain with JSON inside it, the parse still works; if the response is HTML or XML, it does not.
The probe returns json_path_multi_match for a single element. A path like $.items[*] matches every element. Use an index ($.items[0]) or a filter ($.items[?(@.name=='checkout')].count) to pick exactly one.
The probe returns json_path_non_numeric on a value that looks numeric. The matched string may contain units or separators ("42ms", "1,024"). Test the path manually with jq or jsonpath-plus against the literal response body.

Secret headers

Use header_refs for any header that carries a credential: bearer tokens, API keys, session cookies. Each entry maps a header name to the name of an environment variable on the agent host. The agent reads the value at probe time and sends it with the request. Observer stores only the variable name, never the token.

{
  "url": "https://api.example.com/internal/health",
  "headers": { "Accept": "application/json" },
  "header_refs": {
    "Authorization": "OBSERVER_API_TOKEN",
    "X-Api-Key": "OBSERVER_VENDOR_KEY"
  }
}

Set the variables on the agent process (Docker -e, a Compose environment: block, or a Kubernetes secret mapped to env) and restart the agent:

export OBSERVER_API_TOKEN="Bearer eyJhbGciOi..."
export OBSERVER_VENDOR_KEY="k_live_..."

The env var holds the full header value, so include any scheme prefix such as Bearer yourself. Rules:

Header names must be valid HTTP header names. Variable names must be UPPER_SNAKE_CASE. Up to 32 entries.
A header can be in headers or header_refs, not both.
Secret headers are merged over headers and follow the same redirect rule: they are sent on same-origin redirects and dropped when a redirect leaves the original origin.
If a variable is unset or blank the probe reports no_data with header_ref_missing and does not send the request. The result names the header and the variable, never the value.
The agent never logs secret header values or includes them in results.

In the metric form, add a header row and turn on Secret, then enter the variable name instead of a value. The form’s Test extraction button for json_path runs the check on the metric’s agent, with its secret headers, when that agent is online and runs 1.7.0 or later. Otherwise it runs from Observer’s servers, which cannot read agent env vars, so secret headers are not sent.

Agent versionheader_refs requires agent 1.6.0 or later. Agents at 1.5.0 or older do not recognise the field and report invalid_config for any probe that uses it. Upgrade the agent before adding secret headers.

mTLS authentication

Set the three *_ref fields to monitor an HTTPS endpoint that requires a client certificate (service meshes, zero-trust networks, internal APIs with cert-based auth).

When not to use mTLS

mTLS is only for endpoints that demand a client certificate. For ordinary HTTPS, including endpoints with a self-signed or private-CA server cert, leave the *_ref fields blank:

Standard public HTTPS: nothing to configure; verify_tls defaults on.
Self-signed server cert you don’t want to verify: set verify_tls to false (don’t reach for mTLS).
Private-CA server cert you do want to verify, but no client cert is required: that’s a server-trust concern, not mTLS. Point the agent’s system trust store at your CA, or use a probe without verification.

How certificates are stored

The cloud never sees your certificate or private key. Each *_ref field holds the name of an environment variable on the agent host. The agent reads the PEM material from that variable at probe time. The value can be either:

the PEM text directly (handy for Docker -e / Compose env), or
a filesystem path to a PEM file (handy for Kubernetes secret mounts).

Example agent environment:

# PEM inline
export OBSERVER_MTLS_CLIENT_CERT="$(cat client.crt)"
export OBSERVER_MTLS_CLIENT_KEY="$(cat client.key)"

# or a path the agent can read
export OBSERVER_MTLS_CLIENT_CERT=/etc/observer/mtls/client.crt
export OBSERVER_MTLS_CLIENT_KEY=/etc/observer/mtls/client.key
export OBSERVER_MTLS_CA_CERT=/etc/observer/mtls/ca.crt   # optional

Then in the metric’s HTTP form, open mTLS authentication and set:

Client certificate env var: OBSERVER_MTLS_CLIENT_CERT
Client private key env var: OBSERVER_MTLS_CLIENT_KEY
CA certificate env var (optional): OBSERVER_MTLS_CA_CERT

The private key never leaves the agent host and is never logged.

Generating a client certificate

Your service operator issues client certs from the mesh / internal CA. For a quick local test against an mTLS server:

# client key + CSR (request the clientAuth EKU; many servers reject
# a client cert that lacks it)
openssl req -newkey rsa:4096 -nodes -keyout client.key -out client.csr \
  -subj "/CN=observer-probe" \
  -addext "extendedKeyUsage=clientAuth"
# sign with your CA, carrying the clientAuth EKU into the cert
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365 -copy_extensions copy

RSA keys must be at least 2048 bits (the example uses 4096 for headroom); weaker keys are rejected at probe time with mtls_weak_key. EC keys (openssl ecparam -genkey) are also accepted, with no bit-floor caveat. If the server rejects the cert with mtls_client_cert_rejected, the most common cause is a missing clientAuth extended-key-usage.

Certificate expiry

The agent parses the client cert on every probe. While the cert is still valid, the probe succeeds and nothing about its expiry is reported to Observer, so track the expiry date yourself (for example openssl x509 -enddate -noout -in client.crt on the agent host). Once the cert is actually expired, the probe returns no_data with reason mtls_cert_expired.

mTLS reason codes

Reason	Meaning
`mtls_incomplete`	Only one of cert / key was configured. Set both.
`mtls_ref_missing`	The named env var isn’t set on the agent.
`mtls_file_unreadable`	The ref points at a path the agent can’t read.
`mtls_cert_parse_failed`	The cert PEM didn’t parse.
`mtls_key_parse_failed`	The key PEM didn’t parse.
`mtls_weak_key`	RSA key under 2048 bits.
`mtls_cert_expired`	The client cert’s validity has passed.
`mtls_server_verify_failed`	The server cert didn’t validate against the CA / trust store. Set `ca_cert_ref` if the server uses a private CA.
`mtls_server_cert_expired`	The server’s TLS cert has expired (server-side problem).
`mtls_client_cert_rejected`	The handshake failed: the server didn’t accept the client cert. Confirm the cert chains to a CA the server trusts and isn’t revoked.
`mtls_handshake_failed`	The request failed with an error that matches no other `mtls_*` code and carries no error code of its own.

Troubleshooting

mtls_client_cert_rejected but the cert looks valid. The server decides which CAs to trust for client auth. Confirm your client cert chains to a CA in the server’s client-CA list, and that the cert isn’t revoked.
mtls_server_verify_failed against an internal server. The server presents a cert signed by a private CA the agent’s system trust store doesn’t know. Set ca_cert_ref to a CA bundle env var.
mtls_ref_missing after setting the variable. The agent reads process.env at probe time. Set the variable on the agent process / pod, not just your shell, and restart the agent.

Quickstart

Concepts

Guides

Reference

Configure HTTP probes

Configuration shape

Field reference

Reason codes

Threshold examples

JSON path extraction

When to use it

When not to use it

Path syntax

Value rules

Reason codes specific to JSON path

Troubleshooting

Secret headers

mTLS authentication

When not to use mTLS

How certificates are stored

Generating a client certificate

Certificate expiry

mTLS reason codes

Troubleshooting

​Configuration shape

​Field reference

​Reason codes

​Threshold examples

​JSON path extraction

​When to use it

​When not to use it

​Path syntax

​Value rules

​Reason codes specific to JSON path

​Troubleshooting

​Secret headers

​mTLS authentication

​When not to use mTLS

​How certificates are stored

​Generating a client certificate

​Certificate expiry

​mTLS reason codes

​Troubleshooting

Configuration shape

Field reference

Reason codes

Threshold examples

JSON path extraction

When to use it

When not to use it

Path syntax

Value rules

Reason codes specific to JSON path

Troubleshooting

Secret headers

mTLS authentication

When not to use mTLS

How certificates are stored

Generating a client certificate

Certificate expiry

mTLS reason codes

Troubleshooting