> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Receive metrics over OpenTelemetry (OTLP)

> Run the agent as an OTLP/HTTP receiver so any OpenTelemetry-compatible sender can push metrics into Observer.

The agent runs an HTTP server on port 4318 that accepts OpenTelemetry
metric pushes. Any OTLP-compatible sender (OpenTelemetry SDK,
OpenTelemetry Collector, vendor agent with an OTLP exporter)
writes metrics to it; each metric you configure in Observer becomes a
probe you can threshold on.

This is the only push-mode source. Every other source type
(`prometheus`, `http`, `tcp`, etc.) pulls on a fixed interval. OTLP
inverts the direction: the sender drives.

## Listener configuration

Set on the agent process:

```bash theme={null}
# Bind address. Defaults to loopback so only same-host processes can
# push without an explicit token. Set to 0.0.0.0:4318 to accept pushes
# from other hosts; a bearer token becomes mandatory in that case.
OBSERVER_OTLP_LISTEN_ADDR=127.0.0.1:4318

# Required when binding a non-loopback address. Without it the receiver
# does not start and OTLP metrics report push_source_init_failed.
OBSERVER_OTLP_BEARER_TOKEN=replace-with-a-32-byte-secret

# Max number of distinct streams (metric_name x attribute fingerprint)
# the receiver keeps in memory. Defaults to 1000.
OBSERVER_OTLP_MAX_BUFFER_POINTS=1000

# Skip the receiver entirely. Useful if you only run pull probes.
OBSERVER_OTLP_DISABLE=true
```

The receiver starts lazily the first time a metric definition with
`source_type: otlp` is dispatched. If no such metric is configured,
no port is opened.

## When NOT to use this

* **You already run Prometheus and the metric is exposed there.** Use
  the `prometheus` source instead. It doesn't require wiring up a
  push pipeline and the agent doesn't need to bind a port.
* **You want server-side aggregation across multiple senders.** The
  receiver records one data point per
  `(metric_name, attribute fingerprint)` stream. It does not merge
  samples across senders. Aggregate at your OpenTelemetry Collector
  with a `groupbyattrs` or `transform` processor before exporting.
* **You can't bound the cardinality of your attributes.** The
  receiver caps streams at 1000 by default. A metric with unbounded
  user IDs or request IDs in its attributes will keep evicting
  itself. Strip high-cardinality attributes at the source.

## Configure a metric in the Observer console

Create a new metric, pick **OpenTelemetry (OTLP)** from the source
picker, and fill in:

* **Metric name**: the OTLP `metric.name` field your sender emits (`http.server.duration`, `queue.depth`, etc.). Exact match.
* **Attribute filters** (optional): one `key=value` per line. Only data points carrying every key=value pair are read. Leave blank to read whichever matching data point is most recent.
* **Aggregation**: how to collapse the data point into a single number. See [Aggregation reference](#aggregation-reference).
* **Staleness window**: when the most recent matching sample is older than this, the probe surfaces `no_data` with reason [`otlp_stale`](#reason-codes-specific-to-otlp). Default 120 seconds.

The threshold fields work the same as any other source: the
aggregation produces a scalar; healthy and unhealthy bounds map that
scalar to status.

## Aggregation reference

| Aggregation | Gauge / Sum | Histogram | Exponential histogram |
| - | - | - | - |
| `latest` | the data point value | mean (sum / count) | mean |
| `count` | the data point value | the bucket count total | the count |
| `sum` | the data point value | the sum | the sum |
| `mean` | the data point value | sum / count | sum / count |
| `p50` / `p95` / `p99` | the data point value | linear interpolation across the bucket containing the target rank | linear interpolation across the positive / negative / zero buckets |

For gauges and sums the aggregation field is effectively ignored
because there is only one scalar to read. Pick `latest` to be
explicit.

## Sending from the OpenTelemetry Collector

Add an OTLP/HTTP exporter to your collector config:

```yaml theme={null}
exporters:
  otlphttp/observer:
    endpoint: http://<agent-host>:4318
    encoding: json
    # Required only if the agent has OBSERVER_OTLP_BEARER_TOKEN set.
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/observer]
```

`encoding: json` is the v1 requirement. The default for `otlphttp` is
protobuf; the agent rejects that with HTTP 415 so the collector logs
make the cause obvious.

## Sending from an application

The OpenTelemetry SDKs ship OTLP exporters in every language binding.
JavaScript example:

```ts theme={null}
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";

const exporter = new OTLPMetricExporter({
  url: "http://<agent-host>:4318/v1/metrics",
  headers: { Authorization: "Bearer <your-token>" },
});
```

Equivalent SDKs exist for Python, Go, Java, .NET, and Ruby. Each
follows the same `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS`
environment variable contract.

## Reason codes specific to OTLP

The `reason` field on `no_data` results carries one of:

* `otlp_no_samples`: nothing has arrived with this metric name and
  attribute filter since the agent started. Verify the sender is
  pointed at the right port and `metric_name` matches exactly.
* `otlp_stale`: the most recent matching sample is older than the
  configured staleness window. The sender has stopped pushing, or
  the metric was renamed at the source.
* `otlp_aggregation_unavailable`: the chosen aggregation is not
  extractable from the data point shape (e.g. a non-finite bucket
  base in a malformed exponential histogram). Switch to `mean`,
  `count`, or `sum` and verify the sender's bucket geometry.
* `otlp_receiver_disabled`: the agent was started with
  `OBSERVER_OTLP_DISABLE=true`. Unset that variable and restart.
* `push_source_init_failed`: the receiver could not start. Common
  cause: another process already bound the configured port. Free
  the port or change `OBSERVER_OTLP_LISTEN_ADDR`.

## Security

By default the receiver binds `127.0.0.1` only. Other processes on
the same host can push metrics; the network cannot. With a non-loopback
bind and no `OBSERVER_OTLP_BEARER_TOKEN`, the receiver does not start
and OTLP metrics report `push_source_init_failed`.

Authentication is a constant-time bearer-token compare against the
`Authorization: Bearer <token>` header. Generate the token with 32
bytes of OS randomness (`openssl rand -hex 32`). Rotate by editing
the env var and restarting the agent; the sender must update at the
same time.

The receiver does not inspect TLS. If you need transport encryption
across a network, terminate TLS at a sidecar reverse proxy and have
the proxy forward to the loopback receiver.

## Backpressure and limits

The receiver buffers the most recent data point per stream
(`metric_name` × attribute fingerprint). When the stream count
exceeds `OBSERVER_OTLP_MAX_BUFFER_POINTS`, the oldest stream is
dropped and a counter on the agent dashboard increments. There is no
queue between the receiver and the buffer: each accepted request
writes synchronously. Senders are never blocked.

For high-cardinality metric streams, prefer fewer distinct attribute
combinations per metric or split into multiple metric names. The cap
exists to bound memory growth on stuck-tag-explosion incidents.

## Known limits

These are intentional design decisions, surfaced here so you know
which workloads to route elsewhere.

* **OTLP/gRPC on port 4317 is not implemented.** The receiver returns
  HTTP 415 on `application/x-protobuf`. Configure your sender for
  OTLP/HTTP JSON.
* **OTLP traces and logs are not accepted.** Observer's domain is
  status, and metrics drive status. Traces and logs go to a tracing /
  logging backend.
* **Exponential-histogram quantile estimation is geometry-based.**
  The agent walks the `positive`, `negative`, and zero buckets in
  ascending numeric order using `base = 2 ^ (2 ^ -scale)` and
  interpolates within the target bucket. Accuracy depends on the
  sender's `scale` (higher scale = finer buckets = tighter
  estimate). Senders that emit only a few buckets per decade trade
  storage for precision; pick `mean` if the rough center is enough.

## Troubleshooting

Each entry leads with the symptom in bold and the action to take.

* **`otlp_no_samples` while the collector logs successful exports.**
  Verify the metric name is byte-for-byte identical between your
  sender and the Observer metric definition. OTLP metric names are
  case-sensitive and `.` is significant. The collector's `debug`
  exporter (set `verbosity: detailed`) prints the names it received.
* **Sender gets `415 Unsupported Media Type`.** Your exporter is
  using protobuf. Set `encoding: json` on the `otlphttp` exporter
  (collector) or send the SDK to the JSON exporter package.
* **Sender gets `Connection refused`.** The receiver only opens its
  port after the first OTLP metric definition is dispatched. In the
  console, create a metric with source type **OpenTelemetry (OTLP)**
  and save it, then retry after the agent picks up the definition (usually within
  about a minute).
* **Receiver runs but no metrics show up in Observer.** From the
  agent host: `ss -tlnp | grep 4318` should list the agent process
  bound to the configured address. If it lists `127.0.0.1`, a
  sender on another host can't reach it. Set
  `OBSERVER_OTLP_LISTEN_ADDR=0.0.0.0:4318`, set
  `OBSERVER_OTLP_BEARER_TOKEN`, restart the agent, and add
  `Authorization: Bearer <token>` to the sender.
* **`otlp_aggregation_unavailable` on a quantile (p50/p95/p99).**
  Most often a malformed exponential histogram: the sender shipped
  a `scale` outside \[-10, 20], or the bucket geometry produced
  non-finite bounds. Verify the sender's exporter configuration and
  the `scale` it picks; switch to `mean`, `count`, or `sum` if the
  geometry is fixed at the source.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.