> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure HTTP probes

> Probe an HTTP endpoint and report response time as the metric value.

HTTP probes hit a URL on the configured interval. The reported
value is `response_time_ms` for successful requests, or `no_data`
with a reason code when the request fails (timeout, connection
refused, body mismatch, unexpected status).

## Configuration shape

```json theme={null}
{
  "url": "https://api.example.com/healthz",
  "method": "GET",
  "expected_status": 200,
  "timeout_ms": 5000,
  "headers": { "User-Agent": "observer-agent" },
  "header_refs": { "Authorization": "OBSERVER_API_TOKEN" },
  "body_match": "ok",
  "follow_redirects": true,
  "verify_tls": true
}
```

## Field reference

| Field | Default | Notes |
| - | - | - |
| `url` | required | Full http(s) URL. URLs with embedded credentials (`user:pass@`) are rejected; use `header_refs`. |
| `method` | `GET` | One of `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`. |
| `expected_status` | `200` | Single integer or array. The probe matches if the response code is in the set. |
| `timeout_ms` | `5000` | Aborts the request when exceeded. Reports `ETIMEDOUT`. |
| `headers` | none | Extra non-secret request headers, stored as entered. Names that look like credentials, matched case-insensitively (`Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`, `X-API-Key`, `Api-Key`, `ApiKey`, anything containing `token`, `secret`, `password`, `passwd`, `auth`, or `session`, or ending in `key`), are rejected here; use `header_refs`. |
| `header_refs` | none | Secret request headers. Maps a header name to the **name** of an env var on the agent host; the agent sends that env var's value. Agent 1.6.0 and later. See [Secret headers](#secret-headers). |
| `body_match` | none | Optional substring match against the first 4KB of the response body. Mismatch reports `body_mismatch`. |
| `follow_redirects` | `true` | When `false`, redirect responses count against `expected_status`. |
| `verify_tls` | `true` | When `false`, the probe accepts invalid TLS certificates. Useful for self-signed internal endpoints. |
| `json_path` | none | Optional JSONPath expression. When set, the probe value is the number extracted from the JSON response at this path, not the response time. See [JSON path extraction](#json-path-extraction). |
| `client_cert_ref` | none | Env-var name on the agent holding the client certificate PEM (or a path to it). Enables mTLS. See [mTLS authentication](#mtls-authentication). |
| `client_key_ref` | none | Env-var name holding the client private key PEM (or a path to it). Required with `client_cert_ref`. See [mTLS authentication](#mtls-authentication). |
| `ca_cert_ref` | none | Optional env-var name holding a CA cert PEM (or a path to one) to verify the server. When set, the probe trusts only that CA; when unset, the system trust store. Works with or without mTLS from agent 1.6.1; earlier versions only read it together with a client certificate. |

## Reason codes

The full catalog, with the hint the console shows for each code, is in [Reason codes](https://docs.use.observer/agent/reference/reason-codes.md).

The `reason` field on `no_data` results is a stable code. The agent
normalises network errors from its runtime into one set, shared with
the TCP, DNS, TLS certificate and Prometheus sources:

* `ETIMEDOUT`: request exceeded `timeout_ms`.
* `ECONNREFUSED`: the host resolved but refused the connection (port
  closed, service down, or a firewall rejecting it).
* `ENOTFOUND`, `EAI_AGAIN`: DNS resolution failed.
* `ECONNRESET`: the target closed the connection before answering.
* `EHOSTUNREACH`: no network route from the agent to the target.
* `connect_failed`: the connection could not be opened for another
  reason.
* `CERT_HAS_EXPIRED`, `DEPTH_ZERO_SELF_SIGNED_CERT`,
  `ERR_TLS_CERT_ALTNAME_INVALID` and the other certificate codes, or
  `tls_handshake_failed`: the TLS handshake failed.
* `ConnectionRefused`: reported by older agents for both a refused
  port and a hostname that does not resolve. The console shows it as
  **Could not connect**. Current agents report `ECONNREFUSED` or
  `ENOTFOUND` instead.
* `invalid_url`: `url` could not be parsed.
* `invalid_url_scheme`: `url` uses a scheme other than `http` or
  `https`.
* `http_error`: the request failed without a more specific error
  code.
* `unexpected_status:<code>`: status code not in `expected_status`.
* `body_mismatch`: `body_match` was set and the response body did
  not contain it.
* `header_ref_missing`: an env var named in `header_refs` is not set
  (or is blank) on the agent host. The request is not sent.
* `header_ref_invalid`: an env var named in `header_refs` holds a line
  break or control character, which is not allowed in a header.

> [!NOTE]
>
> **Body match limit**
>
> Only the first 4KB of the response body is read. If the marker
> string is later in the response, the probe reports
> `body_mismatch`. Move the marker earlier in the response, or use
> a dedicated health endpoint that returns it in the first kilobyte.

## Threshold examples

| Goal | Healthy | Unhealthy |
| - | - | - |
| Endpoint reachable, fast | `under 500` | `over 2000` |
| Endpoint reachable | `under 5000` | `over 10000` |

For pure reachability with no latency requirement, set the
unhealthy threshold equal to the timeout and rely on `no_data`
for failures.

## JSON path extraction

Set `json_path` when the value worth thresholding lives inside the
response body, not in the response time. The agent fetches the URL,
parses the body as JSON, applies the JSONPath expression, and uses
the extracted value as the probe value.

```json theme={null}
{
  "url": "https://api.example.com/internal/queues",
  "method": "GET",
  "expected_status": 200,
  "json_path": "$.pending_count"
}
```

If `pending_count` is `42` in the response, the probe value is `42`.
Threshold on it like any other numeric probe.

### When to use it

* A custom health endpoint that returns a payload like
  `{"status": "ok", "queue_depth": 42}` and you want to alert on
  queue depth, not just status code.
* A third-party JSON API where the field you care about is buried
  in the response.
* An internal metric endpoint that returns several numbers; pick
  one per probe.

### When not to use it

* The endpoint exposes Prometheus exposition format. Use the
  `prometheus` source type instead, which understands the format
  natively and handles labels.
* The value lives across multiple fields and needs computation
  (sum, ratio, percentile). Compute it at the source and expose
  a single field, or move to a Prometheus-style metric.
* The response is large (multi-MB). The agent caps the body at
  10 MB and reports `json_body_too_large`. Add a dedicated probe
  endpoint instead.

### Path syntax

The agent evaluates paths with the `jsonpath-plus` dialect of JSONPath. Common patterns:

| Path | Matches |
| - | - |
| `$.field` | Top-level field. |
| `$.parent.child` | Nested field. |
| `$.items[0]` | First element of an array. |
| `$.items[-1:]` | Last element of an array. |
| `$.items[?(@.name=='checkout')].count` | Filtered array element. |

### Value rules

The expression must resolve to **exactly one** value. The agent
coerces:

* Numbers are used directly.
* Booleans: `true` becomes `1`, `false` becomes `0`.
* Strings that parse as finite numbers (e.g. `"42"`, `"3.14"`)
  become the parsed number. Surrounding whitespace is ignored.
* Anything else reports `no_data` with a specific reason code (see below).

### Reason codes specific to JSON path

* `json_parse_failed`: response body was not valid JSON.
* `json_path_invalid`: the `json_path` expression could not be
  evaluated. Check its syntax.
* `json_path_no_match`: the path matched nothing in the response.
* `json_path_multi_match`: the path matched more than one value.
  Tighten the expression.
* `json_path_non_numeric`: the matched value was a string that
  could not be parsed as a number, or another non-numeric scalar.
* `json_path_unsupported_type`: the matched value was an array,
  object, or `null`. The path must point at a leaf value.
* `json_body_too_large`: the response exceeded 10 MB. The agent
  caps body reads to protect memory.

> [!NOTE]
>
> **Body match still applies**
>
> `body_match` and `json_path` work together. If both are set, the
> body must contain the substring AND the JSONPath must extract a
> numeric value. Use `body_match` as a coarse sanity check (e.g.
> the response includes the word `ok`) and `json_path` for the
> actual probe value.

### Troubleshooting

* **The probe always returns `json_parse_failed`.** Check the
  endpoint's `Content-Type`. The agent does not require
  `application/json`, but the body must be parseable as JSON. If
  the response is `text/plain` with JSON inside it, the parse
  still works; if the response is HTML or XML, it does not.
* **The probe returns `json_path_multi_match` for a single
  element.** A path like `$.items[*]` matches every element. Use
  an index (`$.items[0]`) or a filter
  (`$.items[?(@.name=='checkout')].count`) to pick exactly one.
* **The probe returns `json_path_non_numeric` on a value that
  looks numeric.** The matched string may contain units or
  separators (`"42ms"`, `"1,024"`). Test the path manually with
  `jq` or `jsonpath-plus` against the literal response body.

## Secret headers

Use `header_refs` for any header that carries a credential: bearer
tokens, API keys, session cookies. Each entry maps a header name to the
**name of an environment variable on the agent host**. The agent reads
the value at probe time and sends it with the request. Observer stores
only the variable name, never the token.

```json theme={null}
{
  "url": "https://api.example.com/internal/health",
  "headers": { "Accept": "application/json" },
  "header_refs": {
    "Authorization": "OBSERVER_API_TOKEN",
    "X-Api-Key": "OBSERVER_VENDOR_KEY"
  }
}
```

Set the variables on the agent process (Docker `-e`, a Compose
`environment:` block, or a Kubernetes secret mapped to env) and restart
the agent:

```bash theme={null}
export OBSERVER_API_TOKEN="Bearer eyJhbGciOi..."
export OBSERVER_VENDOR_KEY="k_live_..."
```

The env var holds the full header value, so include any scheme prefix
such as `Bearer ` yourself.

Rules:

* Header names must be valid HTTP header names. Variable names must be
  `UPPER_SNAKE_CASE`. Up to 32 entries.
* A header can be in `headers` or `header_refs`, not both.
* Secret headers are merged over `headers` and follow the same redirect
  rule: they are sent on same-origin redirects and dropped when a
  redirect leaves the original origin.
* If a variable is unset or blank the probe reports `no_data` with
  `header_ref_missing` and does not send the request. The result names
  the header and the variable, never the value.
* The agent never logs secret header values or includes them in
  results.

In the metric form, add a header row and turn on **Secret**, then
enter the variable name instead of a value. The form's **Test
extraction** button for `json_path` runs the check on the metric's
agent, with its secret headers, when that agent is online and runs
1.7.0 or later. Otherwise it runs from Observer's servers, which cannot
read agent env vars, so secret headers are not sent.

> [!NOTE]
>
> **Agent version**
>
> `header_refs` requires agent 1.6.0 or later. Agents at 1.5.0 or
> older do not recognise the field and report `invalid_config` for any
> probe that uses it. Upgrade the agent before adding secret headers.

## mTLS authentication

Set the three `*_ref` fields to monitor an HTTPS endpoint that requires
a client certificate (service meshes, zero-trust networks, internal
APIs with cert-based auth).

### When not to use mTLS

mTLS is only for endpoints that demand a **client** certificate. For
ordinary HTTPS, including endpoints with a self-signed or private-CA
**server** cert, leave the `*_ref` fields blank:

* Standard public HTTPS: nothing to configure; `verify_tls` defaults on.
* Self-signed server cert you don't want to verify: set `verify_tls`
  to `false` (don't reach for mTLS).
* Private-CA server cert you do want to verify, but no client cert is
  required: that's a server-trust concern, not mTLS. Point the agent's
  system trust store at your CA, or use a probe without verification.

### How certificates are stored

The cloud never sees your certificate or private key. Each `*_ref`
field holds the **name of an environment variable on the agent host**.
The agent reads the PEM material from that variable at probe time. The
value can be either:

* the PEM text directly (handy for Docker `-e` / Compose env), or
* a filesystem path to a PEM file (handy for Kubernetes secret mounts).

Example agent environment:

```bash theme={null}
# PEM inline
export OBSERVER_MTLS_CLIENT_CERT="$(cat client.crt)"
export OBSERVER_MTLS_CLIENT_KEY="$(cat client.key)"

# or a path the agent can read
export OBSERVER_MTLS_CLIENT_CERT=/etc/observer/mtls/client.crt
export OBSERVER_MTLS_CLIENT_KEY=/etc/observer/mtls/client.key
export OBSERVER_MTLS_CA_CERT=/etc/observer/mtls/ca.crt   # optional
```

Then in the metric's HTTP form, open **mTLS authentication** and set:

* Client certificate env var: `OBSERVER_MTLS_CLIENT_CERT`
* Client private key env var: `OBSERVER_MTLS_CLIENT_KEY`
* CA certificate env var (optional): `OBSERVER_MTLS_CA_CERT`

The private key never leaves the agent host and is never logged.

### Generating a client certificate

Your service operator issues client certs from the mesh / internal CA.
For a quick local test against an mTLS server:

```bash theme={null}
# client key + CSR (request the clientAuth EKU; many servers reject
# a client cert that lacks it)
openssl req -newkey rsa:4096 -nodes -keyout client.key -out client.csr \
  -subj "/CN=observer-probe" \
  -addext "extendedKeyUsage=clientAuth"
# sign with your CA, carrying the clientAuth EKU into the cert
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365 -copy_extensions copy
```

RSA keys must be at least 2048 bits (the example uses 4096 for
headroom); weaker keys are rejected at probe time with
`mtls_weak_key`. EC keys (`openssl ecparam -genkey`) are also
accepted, with no bit-floor caveat. If the server rejects the cert
with `mtls_client_cert_rejected`, the most common cause is a missing
`clientAuth` extended-key-usage.

### Certificate expiry

The agent parses the client cert on every probe. While the cert is
still valid, the probe succeeds and nothing about its expiry is
reported to Observer, so track the expiry date yourself (for example
`openssl x509 -enddate -noout -in client.crt` on the agent host). Once
the cert is actually expired, the probe
returns `no_data` with reason `mtls_cert_expired`.

### mTLS reason codes

| Reason | Meaning |
| - | - |
| `mtls_incomplete` | Only one of cert / key was configured. Set both. |
| `mtls_ref_missing` | The named env var isn't set on the agent. |
| `mtls_file_unreadable` | The ref points at a path the agent can't read. |
| `mtls_cert_parse_failed` | The cert PEM didn't parse. |
| `mtls_key_parse_failed` | The key PEM didn't parse. |
| `mtls_weak_key` | RSA key under 2048 bits. |
| `mtls_cert_expired` | The client cert's validity has passed. |
| `mtls_server_verify_failed` | The server cert didn't validate against the CA / trust store. Set `ca_cert_ref` if the server uses a private CA. |
| `mtls_server_cert_expired` | The server's TLS cert has expired (server-side problem). |
| `mtls_client_cert_rejected` | The handshake failed: the server didn't accept the client cert. Confirm the cert chains to a CA the server trusts and isn't revoked. |
| `mtls_handshake_failed` | The request failed with an error that matches no other `mtls_*` code and carries no error code of its own. |

### Troubleshooting

* **`mtls_client_cert_rejected` but the cert looks valid.** The server
  decides which CAs to trust for client auth. Confirm your client cert
  chains to a CA in the server's client-CA list, and that the cert
  isn't revoked.
* **`mtls_server_verify_failed` against an internal server.** The
  server presents a cert signed by a private CA the agent's system
  trust store doesn't know. Set `ca_cert_ref` to a CA bundle env var.
* **`mtls_ref_missing` after setting the variable.** The agent reads
  `process.env` at probe time. Set the variable on the agent process
  / pod, not just your shell, and restart the agent.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.