Observer Docs

gRPC probes call the standard gRPC Health Checking Protocol (grpc.health.v1.Health/Check) on the configured interval and report the serving status as 1 (SERVING) or 0 (NOT_SERVING), or the Check round-trip latency in milliseconds. This is the right probe for gRPC services used in service-to-service communication. It does not invoke arbitrary methods on your service: only the well-known health-check RPC, which most gRPC frameworks expose with a few lines of setup.

What it requires of your service

Your server must implement the standard health service. Most gRPC stacks ship a ready-made implementation:

Go: google.golang.org/grpc/health + grpc_health_v1.RegisterHealthServer.
Java: grpc-services HealthStatusManager.
Node: grpc-health-check.
Python: grpcio-health-checking.

If the server does not implement it, the probe reports grpc_unimplemented rather than a connection error, so you can tell the two apart.

Overall health vs a named service

The health service tracks status per service name, plus an overall server status under the empty name:

Leave Service name blank to check the overall server health. This is the common case.
Set a registered service name (for example my.package.MyService) to check just that service. If the name is not registered, the probe reports grpc_service_unknown.

Transport security

Three modes:

Plaintext (plaintext): no TLS. Use inside a trusted cluster network (h2c).
TLS (tls): server-authenticated TLS. Set the CA certificate env var if the server uses a private CA; leave it blank to use the system trust store.
mTLS (mtls): mutual TLS. The agent presents a client certificate. This reuses the same env-var-reference mechanism as the HTTP probe’s mTLS: each field is the NAME of an environment variable on the agent host whose value is the PEM material (or a path to a PEM file). The cloud stores only the variable name, never the certificate or key.

Set the cert / key env vars on the agent and reference them by name. See the HTTP probe guide’s mTLS section for the env-var patterns (systemd, Docker, Kubernetes secret mounts).

Secret metadata

Use metadata_refs for any metadata entry that carries a credential: bearer tokens, API keys, session cookies. Each entry maps a metadata key to the name of an environment variable on the agent host. The agent reads the value at probe time and sends it with the health check. Observer stores only the variable name, never the token.

{
  "host": "grpc.internal",
  "port": 443,
  "tls_mode": "tls",
  "metadata": { "x-tenant": "acme" },
  "metadata_refs": {
    "authorization": "OBSERVER_GRPC_TOKEN",
    "x-api-key": "OBSERVER_GRPC_API_KEY"
  }
}

Set the variables on the agent process (Docker -e, a Compose environment: block, or a Kubernetes secret mapped to env) and restart the agent:

export OBSERVER_GRPC_TOKEN="Bearer eyJhbGciOi..."
export OBSERVER_GRPC_API_KEY="k_live_..."

The env var holds the full metadata value, so include any scheme prefix such as Bearer yourself. Rules:

Keys use letters, digits, -, _ and .. They are case-insensitive and sent lowercase. Keys starting with grpc- are reserved by gRPC, and binary keys ending in -bin are not supported. Variable names must be UPPER_SNAKE_CASE. Up to 32 entries.
A key can be in metadata or metadata_refs, not both.
Values must be printable ASCII. A value with a line break or other control character reports metadata_ref_invalid.
If a variable is unset or blank the probe reports no_data with metadata_ref_missing and does not send the health check. The result names the key and the variable, never the value.
The agent never logs secret metadata values or includes them in results.

In the metric form, add a metadata row and turn on Secret, then enter the variable name instead of a value.

Agent versionmetadata_refs requires agent 1.6.0 or later. Agents at 1.5.0 or older do not recognise the field and report invalid_config for any probe that uses it. Upgrade the agent before adding secret metadata.

Configuration shape

Plaintext overall-health check:

{
  "host": "grpc.internal",
  "port": 50051,
  "tls_mode": "plaintext",
  "interpretation": "health_state",
  "timeout_ms": 5000
}

mTLS check of a named service with an auth token kept on the agent host (see Secret metadata):

{
  "host": "grpc.internal",
  "port": 443,
  "service": "my.package.MyService",
  "tls_mode": "mtls",
  "client_cert_ref": "OBSERVER_GRPC_CLIENT_CERT",
  "client_key_ref": "OBSERVER_GRPC_CLIENT_KEY",
  "ca_cert_ref": "OBSERVER_GRPC_CA",
  "metadata": { "x-tenant": "acme" },
  "metadata_refs": { "authorization": "OBSERVER_GRPC_TOKEN" },
  "interpretation": "health_state",
  "timeout_ms": 5000
}

Field reference

Field	Default	Notes
`host`	required	Hostname or IP of the gRPC server.
`port`	required	gRPC port (often 50051, or 443 behind TLS).
`service`	`""`	Empty checks overall server health; a name checks one registered service.
`tls_mode`	`plaintext`	`plaintext`, `tls`, or `mtls`.
`client_cert_ref` / `client_key_ref`	none	Env var names for the client cert + key. Required for `mtls`.
`ca_cert_ref`	none	Env var name for a CA cert PEM to verify the server. Optional for `tls` / `mtls`.
`metadata`	none	Non-secret gRPC call metadata, stored as entered. Keys that look like credentials, matched case-insensitively (`authorization`, `proxy-authorization`, `cookie`, `set-cookie`, `x-api-key`, `api-key`, `apikey`, anything containing `token`, `secret`, `password`, `passwd`, `auth`, or `session`, or ending in `key`), are rejected here; use `metadata_refs`.
`metadata_refs`	none	Secret gRPC call metadata. Maps a metadata key to the name of an env var on the agent host; the agent sends that env var’s value. Agent 1.6.0 and later. See Secret metadata.
`timeout_ms`	`5000`	Check deadline, 100 to 30000 ms.
`interpretation`	`health_state`	`health_state` (1 / 0) or `latency` (ms).

Interpretations

Interpretation	Value	Threshold idea
`health_state`	`1` for SERVING, `0` for NOT_SERVING. UNKNOWN or an unregistered service is no_data.	Healthy: over 0, Unhealthy: under 1
`latency`	Check round-trip in ms (reported on any successful Check).	Healthy: under 200, Unhealthy: over 1000

Reason codes

grpc_unimplemented: the server doesn’t implement the health service. Add grpc.health.v1.Health to the server.
grpc_service_unknown: the named service isn’t registered. Check the name, or leave it blank for overall health.
grpc_health_unknown: the server answered UNKNOWN (often mid-startup).
grpc_unavailable: connection failed (refused, DNS, network). Distinct from auth and TLS errors.
grpc_timeout: the Check didn’t complete within the timeout.
grpc_unauthenticated: the server rejected the call as unauthenticated. Check the secret metadata and the env var it names.
grpc_permission_denied: authenticated but not authorized.
grpc_tls_failed: TLS handshake failed. Check the TLS mode and set the CA cert env var for a private CA.
grpc_ca_unreadable: the CA cert env var is unset or points at an unreadable file on the agent.
grpc_metadata_invalid: a metadata key or value is not legal gRPC metadata. The health check is not sent.
mtls_*: with tls_mode set to mtls, client certificate and key problems use the same mtls_* codes as HTTP probes (for example mtls_ref_missing or mtls_key_parse_failed). See mTLS authentication.
grpc_error: an uncategorised gRPC error. Check the agent log for the status code.
metadata_ref_missing: an env var named in metadata_refs is not set (or is blank) on the agent host. The health check is not sent.
metadata_ref_invalid: an env var named in metadata_refs holds a line break, control character, or non-ASCII character, which gRPC metadata does not allow.

Troubleshooting

grpc_unimplemented on a service you know is up. The server is reachable but doesn’t register the health service. This is a server change, not a probe setting.
grpc_unavailable only over TLS. If plaintext works and TLS doesn’t, the listener may not be terminating TLS on that port, or the port differs. Confirm the TLS port.
grpc_tls_failed against a private CA. Set ca_cert_ref to an env var holding the CA PEM so the agent can verify the server.
mTLS reports mtls_ref_missing. The field wants the NAME of an env var, not the certificate text. Set the env var on the agent and reference it by name.

Quickstart

Concepts

Guides

Reference

Configure gRPC health-check probes

What it requires of your service

Overall health vs a named service

Transport security

Secret metadata

Configuration shape

Field reference

Interpretations

Reason codes

Troubleshooting

​What it requires of your service

​Overall health vs a named service

​Transport security

​Secret metadata

​Configuration shape

​Field reference

​Interpretations

​Reason codes

​Troubleshooting

What it requires of your service

Overall health vs a named service

Transport security

Secret metadata

Configuration shape

Field reference

Interpretations

Reason codes

Troubleshooting