Receive metrics over OpenTelemetry (OTLP)

The agent runs an HTTP server on port 4318 that accepts OpenTelemetry metric pushes. Any OTLP-compatible sender (OpenTelemetry SDK, OpenTelemetry Collector, vendor agent with an OTLP exporter) writes metrics to it; each metric you configure in Observer becomes a probe you can threshold on.

This is the only push-mode source. Every other source type (prometheus, http, tcp, etc.) pulls on a fixed interval. OTLP inverts the direction: the sender drives.

Listener configuration

Set on the agent process:

# Bind address. Defaults to loopback so only same-host processes can
# push without an explicit token. Set to 0.0.0.0:4318 to accept pushes
# from other hosts; a bearer token becomes mandatory in that case.
OBSERVER_OTLP_LISTEN_ADDR=127.0.0.1:4318

# Required when binding a non-loopback address. The agent refuses to
# start otherwise.
OBSERVER_OTLP_BEARER_TOKEN=replace-with-a-32-byte-secret

# Max number of distinct streams (metric_name x attribute fingerprint)
# the receiver keeps in memory. Defaults to 1000.
OBSERVER_OTLP_MAX_BUFFER_POINTS=1000

# Skip the receiver entirely. Useful if you only run pull probes.
OBSERVER_OTLP_DISABLE=true

The receiver starts lazily the first time a metric definition with source_type: otlp is dispatched. If no such metric is configured, no port is opened.

When NOT to use this

• You already run Prometheus and the metric is exposed there. Use the prometheus source instead. It doesn't require wiring up a push pipeline and the agent doesn't need to bind a port.
• You want server-side aggregation across multiple senders. The receiver records one data point per (metric_name, attribute fingerprint) stream. It does not merge samples across senders. Aggregate at your OpenTelemetry Collector with a groupbyattrs or transform processor before exporting.
• You can't bound the cardinality of your attributes. The receiver caps streams at 1000 by default. A metric with unbounded user IDs or request IDs in its attributes will keep evicting itself. Strip high-cardinality attributes at the source.

Configure a metric in the Observer console

Create a new metric, pick OpenTelemetry (OTLP) from the source picker, and fill in:

• Metric name: the OTLP metric.name field your sender emits (http.server.duration, queue.depth, etc.). Exact match.
• Attribute filters (optional): one key=value per line. Only data points carrying every key=value pair are read. Leave blank to read whichever matching data point is most recent.
• Aggregation: how to collapse the data point into a single number. See Aggregation reference.
• Staleness window: when the most recent matching sample is older than this, the probe surfaces no_data with reason otlp_stale. Default 120 seconds.

The threshold fields work the same as any other source: the aggregation produces a scalar; healthy and unhealthy bounds map that scalar to status.

Aggregation reference

Aggregation	Gauge / Sum	Histogram	Exponential histogram
latest	the data point value	mean (sum / count)	mean
count	the data point value	the bucket count total	the count
sum	the data point value	the sum	the sum
mean	the data point value	sum / count	sum / count
p50 / p95 / p99	the data point value	linear interpolation across the bucket containing the target rank	linear interpolation across the positive / negative / zero buckets

For gauges and sums the aggregation field is effectively ignored because there is only one scalar to read. Pick latest to be explicit.

Sending from the OpenTelemetry Collector

Add an OTLP/HTTP exporter to your collector config:

exporters:
  otlphttp/observer:
    endpoint: http://<agent-host>:4318
    encoding: json
    # Required only if the agent has OBSERVER_OTLP_BEARER_TOKEN set.
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/observer]

encoding: json is the v1 requirement. The default for otlphttp is protobuf; the agent rejects that with HTTP 415 so the collector logs make the cause obvious.

Sending from an application

The OpenTelemetry SDKs ship OTLP exporters in every language binding. JavaScript example:

const exporter = new OTLPMetricExporter({
  url: "http://<agent-host>:4318/v1/metrics",
  headers: { Authorization: "Bearer <your-token>" },
});

Equivalent SDKs exist for Python, Go, Java, .NET, and Ruby. Each follows the same OTLP_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS environment variable contract.

Reason codes specific to OTLP

The reason field on no_data results carries one of:

• otlp_no_samples: nothing has arrived with this metric name and attribute filter since the agent started. Verify the sender is pointed at the right port and metric_name matches exactly.
• otlp_stale: the most recent matching sample is older than the configured staleness window. The sender has stopped pushing, or the metric was renamed at the source.
• otlp_aggregation_unavailable: the chosen aggregation is not extractable from the data point shape (e.g. a non-finite bucket base in a malformed exponential histogram). Switch to mean, count, or sum and verify the sender's bucket geometry.
• otlp_receiver_disabled: the agent was started with OBSERVER_OTLP_DISABLE=true. Unset that variable and restart.
• push_source_init_failed: the receiver could not start. Common cause: another process already bound the configured port. Free the port or change OBSERVER_OTLP_LISTEN_ADDR.

Security

By default the receiver binds 127.0.0.1 only. Other processes on the same host can push metrics; the network cannot. The agent refuses to start with a non-loopback bind unless OBSERVER_OTLP_BEARER_TOKEN is set.

Authentication is a constant-time bearer-token compare against the Authorization: Bearer <token> header. Generate the token with 32 bytes of OS randomness (openssl rand -hex 32). Rotate by editing the env var and restarting the agent; the sender must update at the same time.

The receiver does not inspect TLS. If you need transport encryption across a network, terminate TLS at a sidecar reverse proxy and have the proxy forward to the loopback receiver. mTLS at the agent is a follow-up item.

Backpressure and limits

The receiver buffers the most recent data point per stream (metric_name × attribute fingerprint). When the stream count exceeds OBSERVER_OTLP_MAX_BUFFER_POINTS, the oldest stream is dropped and a counter on the agent dashboard increments. There is no queue between the receiver and the buffer: each accepted request writes synchronously. Senders are never blocked.

For high-cardinality metric streams, prefer fewer distinct attribute combinations per metric or split into multiple metric names. The cap exists to bound memory growth on stuck-tag-explosion incidents.

Known limits

These are intentional design decisions, surfaced here so you know which workloads to route elsewhere.

• OTLP/gRPC on port 4317 is not implemented. The receiver returns HTTP 415 on application/x-protobuf. Configure your sender for OTLP/HTTP JSON.
• OTLP traces and logs are not accepted. Observer's domain is status, and metrics drive status. Traces and logs go to a tracing / logging backend.
• Exponential-histogram quantile estimation is geometry-based. The agent walks the positive, negative, and zero buckets in ascending numeric order using base = 2 ^ (2 ^ -scale) and interpolates within the target bucket. Accuracy depends on the sender's scale (higher scale = finer buckets = tighter estimate). Senders that emit only a few buckets per decade trade storage for precision; pick mean if the rough centre is enough.

Troubleshooting

Each entry leads with the symptom in bold and the action to take.

• otlp_no_samples while the collector logs successful exports. Verify the metric name is byte-for-byte identical between your sender and the Observer metric definition. OTLP metric names are case-sensitive and . is significant. The collector's debug exporter (set verbosity: detailed) prints the names it received.
• Sender gets 415 Unsupported Media Type. Your exporter is using protobuf. Set encoding: json on the otlphttp exporter (collector) or send the SDK to the JSON exporter package.
• Sender gets Connection refused. The receiver only opens its port after the first OTLP metric definition is dispatched. In the console, create a metric with source type OpenTelemetry (OTLP) and save it, then retry the send within five seconds.
• Receiver runs but no metrics show up in Observer. From the agent host: ss -tlnp | grep 4318 should list the agent process bound to the configured address. If it lists 127.0.0.1, a sender on another host can't reach it. Set OBSERVER_OTLP_LISTEN_ADDR=0.0.0.0:4318, set OBSERVER_OTLP_BEARER_TOKEN, restart the agent, and add Authorization: Bearer <token> to the sender.
• otlp_aggregation_unavailable on a quantile (p50/p95/p99). Most often a malformed exponential histogram: the sender shipped a scale outside [-10, 20], or the bucket geometry produced non-finite bounds. Verify the sender's exporter configuration and the scale it picks; switch to mean, count, or sum if the geometry is fixed at the source.