Receive metrics over OpenTelemetry (OTLP)

​

Listener configuration

# Bind address. Defaults to loopback so only same-host processes can
# push without an explicit token. Set to 0.0.0.0:4318 to accept pushes
# from other hosts; a bearer token becomes mandatory in that case.
OBSERVER_OTLP_LISTEN_ADDR=127.0.0.1:4318

# Required when binding a non-loopback address. Without it the receiver
# does not start and OTLP metrics report push_source_init_failed.
OBSERVER_OTLP_BEARER_TOKEN=replace-with-a-32-byte-secret

# Max number of distinct streams (metric_name x attribute fingerprint)
# the receiver keeps in memory. Defaults to 1000.
OBSERVER_OTLP_MAX_BUFFER_POINTS=1000

# Skip the receiver entirely. Useful if you only run pull probes.
OBSERVER_OTLP_DISABLE=true

​

When NOT to use this

• You already run Prometheus and the metric is exposed there. Use the prometheus source instead. It doesn’t require wiring up a push pipeline and the agent doesn’t need to bind a port.
• You want server-side aggregation across multiple senders. The receiver records one data point per (metric_name, attribute fingerprint) stream. It does not merge samples across senders. Aggregate at your OpenTelemetry Collector with a groupbyattrs or transform processor before exporting.
• You can’t bound the cardinality of your attributes. The receiver caps streams at 1000 by default. A metric with unbounded user IDs or request IDs in its attributes will keep evicting itself. Strip high-cardinality attributes at the source.

​

Configure a metric in the Observer console

• Metric name: the OTLP metric.name field your sender emits (http.server.duration, queue.depth, etc.). Exact match.
• Attribute filters (optional): one key=value per line. Only data points carrying every key=value pair are read. Leave blank to read whichever matching data point is most recent.
• Aggregation: how to collapse the data point into a single number. See Aggregation reference.
• Staleness window: when the most recent matching sample is older than this, the probe surfaces no_data with reason otlp_stale. Default 120 seconds.

​

Aggregation reference

​

Sending from the OpenTelemetry Collector

exporters:
  otlphttp/observer:
    endpoint: http://<agent-host>:4318
    encoding: json
    # Required only if the agent has OBSERVER_OTLP_BEARER_TOKEN set.
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/observer]

​

Sending from an application

import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";

const exporter = new OTLPMetricExporter({
  url: "http://<agent-host>:4318/v1/metrics",
  headers: { Authorization: "Bearer <your-token>" },
});

​

Reason codes specific to OTLP

• otlp_no_samples: nothing has arrived with this metric name and attribute filter since the agent started. Verify the sender is pointed at the right port and metric_name matches exactly.
• otlp_stale: the most recent matching sample is older than the configured staleness window. The sender has stopped pushing, or the metric was renamed at the source.
• otlp_aggregation_unavailable: the chosen aggregation is not extractable from the data point shape (e.g. a non-finite bucket base in a malformed exponential histogram). Switch to mean, count, or sum and verify the sender’s bucket geometry.
• otlp_receiver_disabled: the agent was started with OBSERVER_OTLP_DISABLE=true. Unset that variable and restart.
• push_source_init_failed: the receiver could not start. Common cause: another process already bound the configured port. Free the port or change OBSERVER_OTLP_LISTEN_ADDR.

​

Security

​

Backpressure and limits

​

Known limits

• OTLP/gRPC on port 4317 is not implemented. The receiver returns HTTP 415 on application/x-protobuf. Configure your sender for OTLP/HTTP JSON.
• OTLP traces and logs are not accepted. Observer’s domain is status, and metrics drive status. Traces and logs go to a tracing / logging backend.
• Exponential-histogram quantile estimation is geometry-based. The agent walks the positive, negative, and zero buckets in ascending numeric order using base = 2 ^ (2 ^ -scale) and interpolates within the target bucket. Accuracy depends on the sender’s scale (higher scale = finer buckets = tighter estimate). Senders that emit only a few buckets per decade trade storage for precision; pick mean if the rough center is enough.

​

Troubleshooting

• otlp_no_samples while the collector logs successful exports. Verify the metric name is byte-for-byte identical between your sender and the Observer metric definition. OTLP metric names are case-sensitive and . is significant. The collector’s debug exporter (set verbosity: detailed) prints the names it received.
• Sender gets 415 Unsupported Media Type. Your exporter is using protobuf. Set encoding: json on the otlphttp exporter (collector) or send the SDK to the JSON exporter package.
• Sender gets Connection refused. The receiver only opens its port after the first OTLP metric definition is dispatched. In the console, create a metric with source type OpenTelemetry (OTLP) and save it, then retry after the agent picks up the definition (usually within about a minute).
• Receiver runs but no metrics show up in Observer. From the agent host: ss -tlnp | grep 4318 should list the agent process bound to the configured address. If it lists 127.0.0.1, a sender on another host can’t reach it. Set OBSERVER_OTLP_LISTEN_ADDR=0.0.0.0:4318, set OBSERVER_OTLP_BEARER_TOKEN, restart the agent, and add Authorization: Bearer <token> to the sender.
• otlp_aggregation_unavailable on a quantile (p50/p95/p99). Most often a malformed exponential histogram: the sender shipped a scale outside [-10, 20], or the bucket geometry produced non-finite bounds. Verify the sender’s exporter configuration and the scale it picks; switch to mean, count, or sum if the geometry is fixed at the source.