> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Prometheus query metrics

> Define a metric whose value comes from a PromQL query the agent runs against your Prometheus.

Prometheus is the most common metric source. The agent runs the
PromQL query against the Prometheus URL configured at deploy time
and reports the scalar result.

## Configuration shape

A Prometheus metric carries a `source_config` of:

```json theme={null}
{
  "query": "rate(http_requests_total{job=\"checkout-api\",status=~\"5..\"}[5m]) / rate(http_requests_total{job=\"checkout-api\"}[5m])",
  "prometheus_url": "https://prometheus.example/  (optional override)"
}
```

The optional `prometheus_url` overrides the agent's
`PROMETHEUS_SERVER_URL` for this single metric. Use it when one
agent serves multiple Prometheus servers.

## Query requirements

* The query must return a single scalar value. Use aggregation
  (`sum`, `avg`, `rate`, etc.) to collapse vector results.
* Empty results are reported as `no_data`; the metric does not
  flip status until the query produces a value.
* A result with more than one series is reported as `no_data`
  with reason `prom_multiple_series`.
* The agent does not interpret the query content. The cloud's
  push payload contains the precomputed status, not the query
  string.

## Reason codes

The full catalog, with the hint the console shows for each code, is in [Reason codes](https://docs.use.observer/agent/reference/reason-codes.md).

A metric that cannot produce a value reports `no_data` with one of:

* `prometheus_url_missing`: neither `prometheus_url` nor
  `PROMETHEUS_SERVER_URL` is set.
* `invalid_prometheus_url`: the configured URL is not a valid URL.
* `no_data_for_query`: the query returned no series, or a value that
  is not a finite number.
* `prom_multiple_series`: the query returned more than one series.
  Aggregate it down to one.
* `Unauthorized`, `BadQuery`, `PromUpstream`: Prometheus answered
  401 or 403, 400, or 5xx.
* `prometheus_auth_conflict`: a bearer token and basic auth are both
  set, or `PROMETHEUS_HEADERS` repeats a header another variable
  already sets (agent 1.6.2+).
* `prometheus_headers_invalid`: `PROMETHEUS_HEADERS` is neither a
  JSON object nor a `Name: value; Name2: value` list (agent 1.6.2+).
* `ECONNREFUSED`, `ETIMEDOUT`, `ENOTFOUND`, `ECONNRESET`,
  `EHOSTUNREACH`, `connect_failed`: the request did not reach
  Prometheus. Older agents report a failed lookup as `DNS`, and a
  refused port or a failed lookup as `ConnectionRefused`.

## Authentication

When Prometheus requires basic auth, set on the agent:

```bash theme={null}
PROMETHEUS_BASIC_AUTH_ENABLED=true
PROMETHEUS_USERNAME=...
PROMETHEUS_PASSWORD=...
```

These apply to every Prometheus probe the agent runs.

From agent 1.6.2 the agent can also send a bearer token
(`PROMETHEUS_BEARER_TOKEN`), a tenant header
(`PROMETHEUS_TENANT_ID`) and any other header
(`PROMETHEUS_HEADERS`). A bearer token and basic auth cannot be used
together. Header values are never written to the agent log.

> [!NOTE]
>
> **Grafana Cloud**
>
> Grafana Cloud's hosted Prometheus uses basic auth. The
> credentials issued by Grafana for read access drop in here
> unchanged. See
> [Connect to Grafana Cloud](https://docs.use.observer/agent/guides/connect-grafana-cloud.md).

## Grafana Mimir and other Prometheus-compatible backends

Requires agent 1.6.2 or later.

Any backend that serves the Prometheus HTTP API (`/api/v1/query`)
works as a Prometheus source: Grafana Mimir, Cortex, Thanos Query,
VictoriaMetrics and Grafana Cloud. Set `PROMETHEUS_SERVER_URL` to the
base URL of the query API, including any path prefix. The agent keeps
the prefix and appends `/api/v1/query`.

| Backend | Example `PROMETHEUS_SERVER_URL` |
| - | - |
| Grafana Mimir | `https://mimir.example.com/prometheus` |
| Grafana Cloud | `https://prometheus-prod-01-eu-west-0.grafana.net/api/prom` |
| VictoriaMetrics cluster | `http://vmselect:8481/select/0/prometheus` |
| VictoriaMetrics single node, Thanos Query | `http://victoriametrics:8428`, `http://thanos-query:9090` |

### Grafana Mimir with a tenant

Mimir is multi-tenant: each request names its tenant in the
`X-Scope-OrgID` header. Set the tenant with `PROMETHEUS_TENANT_ID`:

```bash theme={null}
PROMETHEUS_SERVER_URL=https://mimir.example.com/prometheus
PROMETHEUS_TENANT_ID=team-a
PROMETHEUS_BASIC_AUTH_ENABLED=false
```

If Mimir runs with multi-tenancy turned off, leave
`PROMETHEUS_TENANT_ID` unset. If a gateway in front of Mimir also
asks for credentials, add basic auth or a bearer token as below.

### Grafana Cloud with basic auth

Grafana Cloud's hosted Prometheus uses basic auth: the username is
the numeric user ID shown on the stack's Prometheus details page and
the password is an access policy token with the `metrics:read`
scope. See [Connect to Grafana Cloud](https://docs.use.observer/agent/guides/connect-grafana-cloud.md).

```bash theme={null}
PROMETHEUS_SERVER_URL=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom
PROMETHEUS_BASIC_AUTH_ENABLED=true
PROMETHEUS_USERNAME=123456
PROMETHEUS_PASSWORD=glc_...
```

### Bearer token

For a backend or gateway that expects `Authorization: Bearer`, set
`PROMETHEUS_BEARER_TOKEN` and turn basic auth off:

```bash theme={null}
PROMETHEUS_SERVER_URL=https://metrics.example.com/prometheus
PROMETHEUS_BASIC_AUTH_ENABLED=false
PROMETHEUS_BEARER_TOKEN=...
```

A bearer token and basic auth cannot be combined. When both are in
effect (basic auth enabled with `PROMETHEUS_PASSWORD` set), the agent
logs an error at startup and Prometheus metrics report `no_data`
with reason `prometheus_auth_conflict`.

### Other headers

`PROMETHEUS_HEADERS` adds any other header, as a JSON object or a
`Name: value; Name2: value` list:

```bash theme={null}
PROMETHEUS_HEADERS='{"X-Api-Key": "..."}'
# or
PROMETHEUS_HEADERS='X-Api-Key: ...; X-Request-Source: observer'
```

Keep the password, the bearer token and `PROMETHEUS_HEADERS` on the
agent host like any other secret. The Add agent screen in the console
writes them into the install command as variables you fill in on the
host, never as values.

## Threshold examples

| Query | Healthy | Unhealthy |
| - | - | - |
| 5xx error ratio (last 5 min) | `under 0.005` | `over 0.02` |
| p95 latency in ms | `under 500` | `over 2000` |
| Queue depth | `under 100` | `over 1000` |
| Replica count | `over 1` | `under 1` |

The strict-comparison rule applies (see
[threshold operators](https://docs.use.observer/docs/reference/threshold-operators.md) in the
Documentation tab).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.