> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure an Elasticsearch or OpenSearch source

> Turn an Elasticsearch / OpenSearch aggregation into a metric. Error counts, latency percentiles, unique users.

The Elasticsearch source runs one search per interval against
`/{index}/_search` with `size:0` and reads a named aggregation's numeric
value as the metric. OpenSearch uses the same search API, so this source
covers both; the flavor setting only changes labels.

Observer is not a log store. The source extracts one number from an
aggregation; it never reads or keeps documents. Same shape as the SQL
and Loki sources: the query returns a number, that number is the metric.

## When to use it

Anything you have indexed in Elasticsearch / OpenSearch and want as a
status signal: error counts from log events, average or p95 request
latency, unique active users, business event rates.

## Query structure

Provide the full search body with a `query` (optional) and an `aggs`
block. Name the aggregation; the source reads its value.

```json theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "error" } },
        { "range": { "@timestamp": { "gte": "now-5m" } } }
      ]
    }
  },
  "aggs": {
    "error_count": { "value_count": { "field": "@timestamp" } }
  }
}
```

Set the aggregation name to `error_count`; the source reads
`aggregations.error_count.value`. The agent always sends `size:0`, so
documents are never returned.

## Supported aggregations

Single-value metric aggregations are read from `.value`:

| Aggregation | Use |
| - | - |
| `value_count` | how many documents matched (error count, event count) |
| `sum` | total of a numeric field |
| `avg` | average (request latency, response size) |
| `min` / `max` | extremes |
| `cardinality` | distinct values (unique users) |

`percentiles` is read from `.values`. Set the **Percentile** field to the
key you want (for example `95.0`):

```json theme={null}
{ "aggs": { "latency_p95": { "percentiles": { "field": "duration_ms", "percents": [95] } } } }
```

The query must resolve to one number. `value_count` over no matches
returns 0; `avg`/`min`/`max` over no matches return null, which the
probe reports as `es_no_data`.

## Configuration shape

```json theme={null}
{
  "base_url": "https://es.internal.example.com:9200",
  "index": "logs-*",
  "query": { "query": {}, "aggs": { "error_count": { "value_count": { "field": "@timestamp" } } } },
  "agg_name": "error_count",
  "flavor": "elasticsearch",
  "auth_mode": "api_key",
  "api_key_ref": "OBSERVER_ES_API_KEY",
  "timeout_ms": 10000
}
```

## Field reference

| Field | Default | Notes |
| - | - | - |
| `base_url` | required | Elasticsearch / OpenSearch HTTP endpoint. |
| `index` | required | Index or pattern. Wildcards + date math allowed (`logs-*`, `events-2026.*`). |
| `query` | required | Full search body containing an `aggs` block. |
| `agg_name` | required | The key under `aggs` to read the value from. |
| `percentile` | none | For a percentiles aggregation: which percentile key (`95.0`). |
| `flavor` | `elasticsearch` | `elasticsearch` or `opensearch`. Same request either way. |
| `auth_mode` | `none` | `none`, `bearer`, `basic`, or `api_key`. |
| `token_ref` | none | Env var NAME holding the bearer token. Required for `bearer`. |
| `username` / `password_ref` | none | Basic auth username + env var NAME of the password. Both required for `basic`. |
| `api_key_ref` | none | Env var NAME holding the base64 `id:api_key`. Required for `api_key`. |
| `timeout_ms` | `10000` | 100 to 60000 ms. |

## Authentication

Auth secrets stay on the agent. You store the NAME of an environment
variable on the agent host; the agent reads the value at query time, so
the credential never reaches the cloud and never appears in logs.

For **Elastic Cloud**, the usual choice is an API key. Create one in
Kibana (Stack Management, API keys); Elastic gives you a base64
`id:api_key` value. Export it on the agent and reference it:

```bash theme={null}
export OBSERVER_ES_API_KEY="VnVhQ2ZHY0JDZGJrU..."   # api_key_ref: OBSERVER_ES_API_KEY
```

The agent sends it as `Authorization: ApiKey <value>`. Bearer and basic
auth work the same way through `token_ref` and `username` +
`password_ref`.

## OpenSearch

OpenSearch is a fork of Elasticsearch with a compatible search API. Set
the flavor to OpenSearch for clarity; the request is identical. Known
differences (security plugin endpoints, version fields) don't affect the
`_search` + aggregations path this source uses. Basic auth and bearer
tokens both work; OpenSearch doesn't use the Elastic API-key format, so
prefer basic auth there.

## Testing the query

Observer cannot reach a private Elasticsearch from the cloud, so there
is no live "run query" button. Validate the query in Kibana Dev Tools
(or OpenSearch Dashboards) against the same cluster, confirm the
aggregation returns a single number, then paste the body in. The first
value arrives on the next agent tick.

## Reason codes

* `es_agg_not_found`: the aggregation name isn't in the response. The
  available names are in the metadata; check the name matches a key
  under `aggs`.
* `es_agg_unsupported`: the named aggregation didn't yield a single
  number. Use a metric aggregation (and set the percentile for
  `percentiles`).
* `es_no_data`: the aggregation value was null (often an empty match
  set on `avg`/`min`/`max`).
* `es_index_not_found`: 404 for the index pattern. Check it exists.
* `es_unreachable`: couldn't connect (refused, DNS, network).
* `es_timeout`: the search didn't finish in time.
* `es_unauthorized`: 401/403. Check auth + the credential env var.
* `es_auth_ref_missing`: the token / password / API-key env var isn't
  set on the agent.
* `es_query_error`: 400/422. The Query DSL is invalid; the parser error
  is in the metadata. Test it in Dev Tools.
* `es_server_error`: 5xx from the cluster. Usually transient.
* `es_error`: any other failure, such as an unexpected HTTP status or
  a network error not listed above.

## Troubleshooting

* **`es_agg_not_found` with the right query.** The `agg_name` must match
  the key under `aggs` exactly. For nested aggregations, name the
  top-level one whose `.value` you want.
* **`es_no_data` on `avg`.** No documents matched. `avg` over an empty
  set is null; if you want 0, use `value_count` or widen the time range.
* **`es_unauthorized` on Elastic Cloud.** Confirm the API key is the
  base64 `id:api_key`, not the raw key, and that the env var is set on
  the agent.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.