> ## Documentation Index
> Fetch the complete documentation index at: https://docs.use.observer/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure gRPC health-check probes

> Probe a gRPC service using the standard gRPC Health Checking Protocol. Reports SERVING / NOT_SERVING or Check latency.

gRPC probes call the standard [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md)
(`grpc.health.v1.Health/Check`) on the configured interval and report
the serving status as 1 (SERVING) or 0 (NOT\_SERVING), or the Check
round-trip latency in milliseconds.

This is the right probe for gRPC services used in service-to-service
communication. It does not invoke arbitrary methods on your service:
only the well-known health-check RPC, which most gRPC frameworks expose
with a few lines of setup.

## What it requires of your service

Your server must implement the standard health service. Most gRPC
stacks ship a ready-made implementation:

* Go: `google.golang.org/grpc/health` + `grpc_health_v1.RegisterHealthServer`.
* Java: `grpc-services` `HealthStatusManager`.
* Node: `grpc-health-check`.
* Python: `grpcio-health-checking`.

If the server does not implement it, the probe reports
`grpc_unimplemented` rather than a connection error, so you can tell the
two apart.

## Overall health vs a named service

The health service tracks status per service name, plus an overall
server status under the empty name:

* Leave **Service name** blank to check the overall server health. This
  is the common case.
* Set a registered service name (for example `my.package.MyService`) to
  check just that service. If the name is not registered, the probe
  reports `grpc_service_unknown`.

## Transport security

Three modes:

* **Plaintext** (`plaintext`): no TLS. Use inside a trusted cluster
  network (h2c).
* **TLS** (`tls`): server-authenticated TLS. Set the CA certificate env
  var if the server uses a private CA; leave it blank to use the
  system trust store.
* **mTLS** (`mtls`): mutual TLS. The agent presents a client
  certificate. This reuses the same env-var-reference mechanism as the
  HTTP probe's mTLS: each field is the NAME of an environment variable
  on the agent host whose value is the PEM material (or a path to a PEM
  file). The cloud stores only the variable name, never the certificate
  or key.

Set the cert / key env vars on the agent and reference them by name. See
the HTTP probe guide's mTLS section for the env-var patterns
(systemd, Docker, Kubernetes secret mounts).

## Secret metadata

Use `metadata_refs` for any metadata entry that carries a credential:
bearer tokens, API keys, session cookies. Each entry maps a metadata key
to the **name of an environment variable on the agent host**. The agent
reads the value at probe time and sends it with the health check.
Observer stores only the variable name, never the token.

```json theme={null}
{
  "host": "grpc.internal",
  "port": 443,
  "tls_mode": "tls",
  "metadata": { "x-tenant": "acme" },
  "metadata_refs": {
    "authorization": "OBSERVER_GRPC_TOKEN",
    "x-api-key": "OBSERVER_GRPC_API_KEY"
  }
}
```

Set the variables on the agent process (Docker `-e`, a Compose
`environment:` block, or a Kubernetes secret mapped to env) and restart
the agent:

```bash theme={null}
export OBSERVER_GRPC_TOKEN="Bearer eyJhbGciOi..."
export OBSERVER_GRPC_API_KEY="k_live_..."
```

The env var holds the full metadata value, so include any scheme prefix
such as `Bearer ` yourself.

Rules:

* Keys use letters, digits, `-`, `_` and `.`. They are case-insensitive
  and sent lowercase. Keys starting with `grpc-` are reserved by gRPC,
  and binary keys ending in `-bin` are not supported. Variable names
  must be `UPPER_SNAKE_CASE`. Up to 32 entries.
* A key can be in `metadata` or `metadata_refs`, not both.
* Values must be printable ASCII. A value with a line break or other
  control character reports `metadata_ref_invalid`.
* If a variable is unset or blank the probe reports `no_data` with
  `metadata_ref_missing` and does not send the health check. The result
  names the key and the variable, never the value.
* The agent never logs secret metadata values or includes them in
  results.

In the metric form, add a metadata row and turn on **Secret**, then
enter the variable name instead of a value.

<Info>
  **Agent version**

  `metadata_refs` requires agent 1.6.0 or later. Agents at 1.5.0 or
  older do not recognise the field and report `invalid_config` for any
  probe that uses it. Upgrade the agent before adding secret metadata.
</Info>

## Configuration shape

Plaintext overall-health check:

```json theme={null}
{
  "host": "grpc.internal",
  "port": 50051,
  "tls_mode": "plaintext",
  "interpretation": "health_state",
  "timeout_ms": 5000
}
```

mTLS check of a named service with an auth token kept on the agent host
(see [Secret metadata](#secret-metadata)):

```json theme={null}
{
  "host": "grpc.internal",
  "port": 443,
  "service": "my.package.MyService",
  "tls_mode": "mtls",
  "client_cert_ref": "OBSERVER_GRPC_CLIENT_CERT",
  "client_key_ref": "OBSERVER_GRPC_CLIENT_KEY",
  "ca_cert_ref": "OBSERVER_GRPC_CA",
  "metadata": { "x-tenant": "acme" },
  "metadata_refs": { "authorization": "OBSERVER_GRPC_TOKEN" },
  "interpretation": "health_state",
  "timeout_ms": 5000
}
```

## Field reference

| Field | Default | Notes |
| - | - | - |
| `host` | required | Hostname or IP of the gRPC server. |
| `port` | required | gRPC port (often 50051, or 443 behind TLS). |
| `service` | `""` | Empty checks overall server health; a name checks one registered service. |
| `tls_mode` | `plaintext` | `plaintext`, `tls`, or `mtls`. |
| `client_cert_ref` / `client_key_ref` | none | Env var names for the client cert + key. Required for `mtls`. |
| `ca_cert_ref` | none | Env var name for a CA cert PEM to verify the server. Optional for `tls` / `mtls`. |
| `metadata` | none | Non-secret gRPC call metadata, stored as entered. Keys that look like credentials, matched case-insensitively (`authorization`, `proxy-authorization`, `cookie`, `set-cookie`, `x-api-key`, `api-key`, `apikey`, anything containing `token`, `secret`, `password`, `passwd`, `auth`, or `session`, or ending in `key`), are rejected here; use `metadata_refs`. |
| `metadata_refs` | none | Secret gRPC call metadata. Maps a metadata key to the **name** of an env var on the agent host; the agent sends that env var's value. Agent 1.6.0 and later. See [Secret metadata](#secret-metadata). |
| `timeout_ms` | `5000` | Check deadline, 100 to 30000 ms. |
| `interpretation` | `health_state` | `health_state` (1 / 0) or `latency` (ms). |

## Interpretations

| Interpretation | Value | Threshold idea |
| - | - | - |
| `health_state` | `1` for SERVING, `0` for NOT\_SERVING. UNKNOWN or an unregistered service is no\_data. | Healthy: over 0, Unhealthy: under 1 |
| `latency` | Check round-trip in ms (reported on any successful Check). | Healthy: under 200, Unhealthy: over 1000 |

## Reason codes

* `grpc_unimplemented`: the server doesn't implement the health service.
  Add `grpc.health.v1.Health` to the server.
* `grpc_service_unknown`: the named service isn't registered. Check the
  name, or leave it blank for overall health.
* `grpc_health_unknown`: the server answered UNKNOWN (often mid-startup).
* `grpc_unavailable`: connection failed (refused, DNS, network).
  Distinct from auth and TLS errors.
* `grpc_timeout`: the Check didn't complete within the timeout.
* `grpc_unauthenticated`: the server rejected the call as
  unauthenticated. Check the secret metadata and the env var it names.
* `grpc_permission_denied`: authenticated but not authorized.
* `grpc_tls_failed`: TLS handshake failed. Check the TLS mode and set the
  CA cert env var for a private CA.
* `grpc_ca_unreadable`: the CA cert env var is unset or points at an
  unreadable file on the agent.
* `grpc_metadata_invalid`: a metadata key or value is not legal gRPC
  metadata. The health check is not sent.
* `mtls_*`: with `tls_mode` set to `mtls`, client certificate and key
  problems use the same `mtls_*` codes as HTTP probes (for example
  `mtls_ref_missing` or `mtls_key_parse_failed`). See
  [mTLS authentication](/agent/guides/http-probes#mtls-authentication).
* `grpc_error`: an uncategorised gRPC error. Check the agent log for the
  status code.
* `metadata_ref_missing`: an env var named in `metadata_refs` is not set
  (or is blank) on the agent host. The health check is not sent.
* `metadata_ref_invalid`: an env var named in `metadata_refs` holds a
  line break, control character, or non-ASCII character, which gRPC
  metadata does not allow.

## Troubleshooting

* **`grpc_unimplemented` on a service you know is up.** The server is
  reachable but doesn't register the health service. This is a server
  change, not a probe setting.
* **`grpc_unavailable` only over TLS.** If plaintext works and TLS
  doesn't, the listener may not be terminating TLS on that port, or the
  port differs. Confirm the TLS port.
* **`grpc_tls_failed` against a private CA.** Set `ca_cert_ref` to an
  env var holding the CA PEM so the agent can verify the server.
* **mTLS reports `mtls_ref_missing`.** The field wants the NAME of an env
  var, not the certificate text. Set the env var on the agent and
  reference it by name.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.