Configure HTTP probes

HTTP probes hit a URL on the configured interval. The reported value is response_time_ms for successful requests, or no_data with a reason code when the request fails (timeout, connection refused, body mismatch, unexpected status).

Configuration shape

{
  "url": "https://api.example.com/healthz",
  "method": "GET",
  "expected_status": 200,
  "timeout_ms": 5000,
  "headers": { "User-Agent": "observer-agent" },
  "body_match": "ok",
  "follow_redirects": true,
  "verify_tls": true
}

Field reference

Field	Default	Notes
`url`	required	Full URL including scheme.
`method`	`GET`	One of `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`.
`expected_status`	`200`	Single integer or array. The probe matches if the response code is in the set.
`timeout_ms`	`5000`	Aborts the request when exceeded. Reports `ETIMEDOUT`.
`headers`	none	Extra request headers. Common use: API key for protected endpoints.
`body_match`	none	Optional substring match against the first 4KB of the response body. Mismatch reports `body_mismatch`.
`follow_redirects`	`true`	When `false`, redirect responses count against `expected_status`.
`verify_tls`	`true`	When `false`, the probe accepts invalid TLS certificates. Useful for self-signed internal endpoints.
`json_path`	none	Optional JSONPath expression. When set, the probe value is the number extracted from the JSON response at this path, not the response time. See JSON path extraction.
`client_cert_ref`	none	Env-var name on the agent holding the client certificate PEM (or a path to it). Enables mTLS. See mTLS authentication.
`client_key_ref`	none	Env-var name holding the client private key PEM (or a path to it). Required with `client_cert_ref`. See mTLS authentication.
`ca_cert_ref`	none	Optional env-var name holding a CA cert PEM to verify the server. Falls back to the system trust store. See mTLS authentication.

Reason codes

The reason field on no_data results uses values from the HTTP client and Node socket layer:

ETIMEDOUT: request exceeded timeout_ms.
ECONNREFUSED: connection refused at the TCP layer.
ENOTFOUND, EAI_AGAIN: DNS resolution failed.
unexpected_status:<code>: status code not in expected_status.
body_mismatch: body_match was set and the response body did not contain it.

Threshold examples

Goal	Healthy	Unhealthy
Endpoint reachable, fast	`under 500`	`over 2000`
Endpoint reachable	`under 5000`	`over 10000`

For pure reachability with no latency requirement, set the unhealthy threshold equal to the timeout and rely on no_data for failures.

JSON path extraction

Set json_path when the value worth thresholding lives inside the response body, not in the response time. The agent fetches the URL, parses the body as JSON, applies the JSONPath expression, and uses the extracted value as the probe value.

{
  "url": "https://api.example.com/internal/queues",
  "method": "GET",
  "expected_status": 200,
  "json_path": "$.pending_count"
}

If pending_count is 42 in the response, the probe value is 42. Threshold on it like any other numeric probe.

When to use it

A custom health endpoint that returns a payload like {"status": "ok", "queue_depth": 42} and you want to alert on queue depth, not just status code.
A third-party JSON API where the field you care about is buried in the response.
An internal metric endpoint that returns several numbers; pick one per probe.

When not to use it

The endpoint exposes Prometheus exposition format. Use the prometheus source type instead, which understands the format natively and handles labels.
The value lives across multiple fields and needs computation (sum, ratio, percentile). Compute it at the source and expose a single field, or move to a Prometheus-style metric.
The response is large (multi-MB). The agent caps the body at 10 MB and reports json_body_too_large. Add a dedicated probe endpoint instead.

Path syntax

The agent uses standard JSONPath (RFC 9535). Common patterns:

Path	Matches
`$.field`	Top-level field.
`$.parent.child`	Nested field.
`$.items[0]`	First element of an array.
`$.items[-1]`	Last element of an array.
`$.items[?(@.name=='checkout')].count`	Filtered array element.

Value rules

The expression must resolve to exactly one value. The agent coerces:

Numbers → used directly.
Booleans → true becomes 1, false becomes 0.
Strings that parse as finite numbers (e.g. "42", "3.14") → the parsed number.
Anything else → no_data with a specific reason code (see below).

Reason codes specific to JSON path

json_parse_failed: response body was not valid JSON.
json_path_no_match: the path matched nothing in the response.
json_path_multi_match: the path matched more than one value. Tighten the expression.
json_path_non_numeric: the matched value was a string that could not be parsed as a number, or another non-numeric scalar.
json_path_unsupported_type: the matched value was an array, object, or null. The path must point at a leaf value.
json_body_too_large: the response exceeded 10 MB. The agent caps body reads to protect memory.

Troubleshooting

The probe always returns json_parse_failed. Check the endpoint's Content-Type. The agent does not require application/json, but the body must be parseable as JSON. If the response is text/plain with JSON inside it, the parse still works; if the response is HTML or XML, it does not.
The probe returns json_path_multi_match for a single element. A path like $.items[*] matches every element. Use an index ($.items[0]) or a filter ($.items[?(@.name=='checkout')].count) to pick exactly one.
The probe returns json_path_non_numeric on a value that looks numeric. The response may carry trailing whitespace or quoted in a way that breaks the parse. Test the path manually with jq or jsonpath-plus against the literal response body.

mTLS authentication

Set the three *_ref fields to monitor an HTTPS endpoint that requires a client certificate (service meshes, zero-trust networks, internal APIs with cert-based auth).

When not to use mTLS

mTLS is only for endpoints that demand a client certificate. For ordinary HTTPS, including endpoints with a self-signed or private-CA server cert, leave the *_ref fields blank:

Standard public HTTPS: nothing to configure; verify_tls defaults on.
Self-signed server cert you don't want to verify: set verify_tls to false (don't reach for mTLS).
Private-CA server cert you do want to verify, but no client cert is required: that's a server-trust concern, not mTLS. Point the agent's system trust store at your CA, or use a probe without verification.

How certificates are stored

The cloud never sees your certificate or private key. Each *_ref field holds the name of an environment variable on the agent host. The agent reads the PEM material from that variable at probe time. The value can be either:

the PEM text directly (handy for Docker -e / Compose env), or
a filesystem path to a PEM file (handy for Kubernetes secret mounts).

Example agent environment:

# PEM inline

# or a path the agent can read

Then in the metric's HTTP form, open mTLS authentication and set:

Client certificate env var: OBSERVER_MTLS_CLIENT_CERT
Client private key env var: OBSERVER_MTLS_CLIENT_KEY
CA certificate env var (optional): OBSERVER_MTLS_CA_CERT

The private key never leaves the agent host and is never logged.

Generating a client certificate

Your service operator issues client certs from the mesh / internal CA. For a quick local test against an mTLS server:

# client key + CSR (request the clientAuth EKU; many servers reject
# a client cert that lacks it)
openssl req -newkey rsa:4096 -nodes -keyout client.key -out client.csr \
  -subj "/CN=observer-probe" \
  -addext "extendedKeyUsage=clientAuth"
# sign with your CA, carrying the clientAuth EKU into the cert
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365 -copy_extensions copy

RSA keys must be at least 2048 bits (the example uses 4096 for headroom); weaker keys are rejected at probe time with mtls_weak_key. EC keys (openssl ecparam -genkey) are also accepted, with no bit-floor caveat. If the server rejects the cert with mtls_client_cert_rejected, the most common cause is a missing clientAuth extended-key-usage.

Certificate expiry

The agent parses the client cert on every probe. When it expires within 30 days, the probe still succeeds but attaches a cert_expiry_warning (with the not_after date and days remaining) to the result metadata. This is not a status flip and not a dashboard reason chip. It rides in the sample's metadata, so wire an alert on it (or check the metric's recent samples) if you want a proactive heads-up. Once the cert is actually expired, the probe returns no_data with reason mtls_cert_expired.

mTLS reason codes

Reason	Meaning
`mtls_incomplete`	Only one of cert / key was configured. Set both.
`mtls_ref_missing`	The named env var isn't set on the agent.
`mtls_file_unreadable`	The ref points at a path the agent can't read.
`mtls_cert_parse_failed`	The cert PEM didn't parse.
`mtls_key_parse_failed`	The key PEM didn't parse.
`mtls_weak_key`	RSA key under 2048 bits.
`mtls_cert_expired`	The client cert's validity has passed.
`mtls_server_verify_failed`	The server cert didn't validate against the CA / trust store. Set `ca_cert_ref` if the server uses a private CA.
`mtls_server_cert_expired`	The server's TLS cert has expired (server-side problem).
`mtls_client_cert_rejected`	The handshake failed: the server didn't accept the client cert. Confirm the cert chains to a CA the server trusts and isn't revoked.

Troubleshooting

mtls_client_cert_rejected but the cert looks valid. The server decides which CAs to trust for client auth. Confirm your client cert chains to a CA in the server's client-CA list, and that the cert isn't revoked.
mtls_server_verify_failed against an internal server. The server presents a cert signed by a private CA the agent's system trust store doesn't know. Set ca_cert_ref to a CA bundle env var.
mtls_ref_missing after setting the variable. The agent reads process.env at probe time. Set the variable on the agent process / pod, not just your shell, and restart the agent.