Receive metrics over OpenTelemetry (OTLP)

The agent runs an HTTP server on port 4318 that accepts OpenTelemetry metric pushes. Any OTLP-compatible sender (OpenTelemetry SDK, OpenTelemetry Collector, vendor agent with an OTLP exporter) writes metrics to it; each metric you configure in Observer becomes a probe you can threshold on.

This is the only push-mode source. Every other source type (prometheus, http, tcp, etc.) pulls on a fixed interval. OTLP inverts the direction: the sender drives.

Listener configuration#

Set on the agent process:

# Bind address. Defaults to loopback so only same-host processes can
# push without an explicit token. Set to 0.0.0.0:4318 to accept pushes
# from other hosts; a bearer token becomes mandatory in that case.
OBSERVER_OTLP_LISTEN_ADDR=127.0.0.1:4318

# Required when binding a non-loopback address. The agent refuses to
# start otherwise.
OBSERVER_OTLP_BEARER_TOKEN=replace-with-a-32-byte-secret

# Max number of distinct streams (metric_name x attribute fingerprint)
# the receiver keeps in memory. Defaults to 1000.
OBSERVER_OTLP_MAX_BUFFER_POINTS=1000

# Skip the receiver entirely. Useful if you only run pull probes.
OBSERVER_OTLP_DISABLE=true

The receiver starts lazily the first time a metric definition with source_type: otlp is dispatched. If no such metric is configured, no port is opened.

When NOT to use this#

• You already run Prometheus and the metric is exposed there. Use the prometheus source instead. It doesn't require wiring up a push pipeline and the agent doesn't need to bind a port.
• You want server-side aggregation across multiple senders. The receiver records one data point per (metric_name, attribute fingerprint) stream. It does not merge samples across senders. Aggregate at your OpenTelemetry Collector with a groupbyattrs or transform processor before exporting.
• You can't bound the cardinality of your attributes. The receiver caps streams at 1000 by default. A metric with unbounded user IDs or request IDs in its attributes will keep evicting itself. Strip high-cardinality attributes at the source.

Configure a metric in the Observer console#

Create a new metric, pick OpenTelemetry (OTLP) from the source picker, and fill in:

• Metric name: the OTLP metric.name field your sender emits (http.server.duration, queue.depth, etc.). Exact match.
• Attribute filters (optional): one key=value per line. Only data points carrying every key=value pair are read. Leave blank to read whichever matching data point is most recent.
• Aggregation: how to collapse the data point into a single number. See Aggregation reference.
• Staleness window: when the most recent matching sample is older than this, the probe surfaces no_data with reason otlp_stale. Default 120 seconds.

The threshold fields work the same as any other source: the aggregation produces a scalar; healthy and unhealthy bounds map that scalar to status.

Aggregation reference#

For gauges and sums the aggregation field is effectively ignored because there is only one scalar to read. Pick latest to be explicit.

Sending from the OpenTelemetry Collector#

Add an OTLP/HTTP exporter to your collector config:

exporters:
  otlphttp/observer:
    endpoint: http://<agent-host>:4318
    encoding: json
    # Required only if the agent has OBSERVER_OTLP_BEARER_TOKEN set.
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/observer]

encoding: json is the v1 requirement. The default for otlphttp is protobuf; the agent rejects that with HTTP 415 so the collector logs make the cause obvious.

Sending from an application#

The OpenTelemetry SDKs ship OTLP exporters in every language binding. JavaScript example:

const exporter = new OTLPMetricExporter({
  url: "http://<agent-host>:4318/v1/metrics",
  headers: { Authorization: "Bearer <your-token>" },
});

Equivalent SDKs exist for Python, Go, Java, .NET, and Ruby. Each follows the same OTLP_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS environment variable contract.

Reason codes specific to OTLP#

The reason field on no_data results carries one of:

• otlp_no_samples: nothing has arrived with this metric name and attribute filter since the agent started. Verify the sender is pointed at the right port and metric_name matches exactly.
• otlp_stale: the most recent matching sample is older than the configured staleness window. The sender has stopped pushing, or the metric was renamed at the source.
• otlp_aggregation_unavailable: the chosen aggregation is not extractable from the data point shape (e.g. a non-finite bucket base in a malformed exponential histogram). Switch to mean, count, or sum and verify the sender's bucket geometry.
• otlp_receiver_disabled: the agent was started with OBSERVER_OTLP_DISABLE=true. Unset that variable and restart.
• push_source_init_failed: the receiver could not start. Common cause: another process already bound the configured port. Free the port or change OBSERVER_OTLP_LISTEN_ADDR.

Security#

By default the receiver binds 127.0.0.1 only. Other processes on the same host can push metrics; the network cannot. The agent refuses to start with a non-loopback bind unless OBSERVER_OTLP_BEARER_TOKEN is set.

Authentication is a constant-time bearer-token compare against the Authorization: Bearer <token> header. Generate the token with 32 bytes of OS randomness (openssl rand -hex 32). Rotate by editing the env var and restarting the agent; the sender must update at the same time.

The receiver does not inspect TLS. If you need transport encryption across a network, terminate TLS at a sidecar reverse proxy and have the proxy forward to the loopback receiver. mTLS at the agent is a follow-up item.

Backpressure and limits#

The receiver buffers the most recent data point per stream (metric_name × attribute fingerprint). When the stream count exceeds OBSERVER_OTLP_MAX_BUFFER_POINTS, the oldest stream is dropped and a counter on the agent dashboard increments. There is no queue between the receiver and the buffer: each accepted request writes synchronously. Senders are never blocked.

For high-cardinality metric streams, prefer fewer distinct attribute combinations per metric or split into multiple metric names. The cap exists to bound memory growth on stuck-tag-explosion incidents.

Known limits#

These are intentional design decisions, surfaced here so you know which workloads to route elsewhere.

• OTLP/gRPC on port 4317 is not implemented. The receiver returns HTTP 415 on application/x-protobuf. Configure your sender for OTLP/HTTP JSON.
• OTLP traces and logs are not accepted. Observer's domain is status, and metrics drive status. Traces and logs go to a tracing / logging backend.
• Exponential-histogram quantile estimation is geometry-based. The agent walks the positive, negative, and zero buckets in ascending numeric order using base = 2 ^ (2 ^ -scale) and interpolates within the target bucket. Accuracy depends on the sender's scale (higher scale = finer buckets = tighter estimate). Senders that emit only a few buckets per decade trade storage for precision; pick mean if the rough center is enough.

Troubleshooting#

Each entry leads with the symptom in bold and the action to take.

• otlp_no_samples while the collector logs successful exports. Verify the metric name is byte-for-byte identical between your sender and the Observer metric definition. OTLP metric names are case-sensitive and . is significant. The collector's debug exporter (set verbosity: detailed) prints the names it received.
• Sender gets 415 Unsupported Media Type. Your exporter is using protobuf. Set encoding: json on the otlphttp exporter (collector) or send the SDK to the JSON exporter package.
• Sender gets Connection refused. The receiver only opens its port after the first OTLP metric definition is dispatched. In the console, create a metric with source type OpenTelemetry (OTLP) and save it, then retry the send within five seconds.
• Receiver runs but no metrics show up in Observer. From the agent host: ss -tlnp | grep 4318 should list the agent process bound to the configured address. If it lists 127.0.0.1, a sender on another host can't reach it. Set OBSERVER_OTLP_LISTEN_ADDR=0.0.0.0:4318, set OBSERVER_OTLP_BEARER_TOKEN, restart the agent, and add Authorization: Bearer <token> to the sender.
• otlp_aggregation_unavailable on a quantile (p50/p95/p99). Most often a malformed exponential histogram: the sender shipped a scale outside [-10, 20], or the bucket geometry produced non-finite bounds. Verify the sender's exporter configuration and the scale it picks; switch to mean, count, or sum if the geometry is fixed at the source.