How to Monitor gRPC Streams in Go

gRPC bidirectional streaming is one of the most frequently misdiagnosed failure modes in Go services. When a bidi stream slows down, engineers often suspect the network or the database first.

The Anatomy of a Bidi Stream That Fails Quietly

A bidirectional stream maintains two message flows simultaneously over a single HTTP/2 connection: the client sends a stream of requests while the server returns a stream of replies. Each side’s write is buffered by the gRPC framework. Writing a value to a stream only hands it to that framework; it does not immediately send it over the network, as the official flow control documentation explains. That handoff is where monitoring begins and where most tuning errors occur.

Flow control prevents a fast sender from overwhelming a slow receiver’s memory. It applies only to streaming RPCs, not unary calls. The gRPC runtime and HTTP/2 manage this by default, which is why problems appear as resource pressure rather than clear errors.

Three transport properties influence every production failure. Streams cannot be load balanced after they start. Each channel connects to an HTTP/2 connection that limits concurrent streams (the performance best practices page clearly states both). Also, gRPC does not provide metrics automatically: OpenTelemetry metrics require opt-in and were introduced when gRPC phased out its older OpenCensus observability path.

The failure pattern is consistent across teams. Message latency increases, CPU usage spikes on one host while the peer remains idle, or memory grows toward the container limit. The root cause is rarely a single bug. It results from a combination of defaults: keepalive disabled on the client, no flow-control awareness in the read loop, and no metric linking a slow consumer to a backed-up sender.

What to Measure: Message Counts Beat Request Counts for Streaming

For unary RPCs, request rate and p99 latency answer most questions. For bidi streams, they do not. A stream that remains open for hours and quietly backpressures appears as one ongoing “request” from Prometheus’s perspective. The important counters are message-level: grpc_server_msg_received_total and grpc_server_msg_sent_total, which the grpc-prometheus tools expose on both server and client sides (confirmed by the OneUptime Prometheus and Grafana monitoring walkthrough). Tracking the rate of change on these counters shows whether messages are actually moving.

The second key metric is bytes handled, because it distinguishes throughput from message count. A stream can send millions of small messages while transferring little data, or deliver a few large ones. grpc.client.attempt.sent_total_compressed_message_size and its received counterpart are histograms, capturing the distribution, not just the total (they are documented in gRPC’s OpenTelemetry metrics reference).

grpc.server.call.duration is a histogram labeled by grpc.method and grpc.status, but interpret it as end-to-end time per RPC from the server transport’s perspective, not per message. It does not effectively diagnose slow stream legs but can detect a handler that hangs. Monitor both layers and keep transport-level latency percentiles separate.

Latency Percentiles, Sampling, and Label Cardinality on Long Streams

Default Prometheus latency histograms are disabled in many gRPC middlewares because of label cardinality. Enable handling-time histograms only after controlling the label set, as the Albert Moreno walkthrough of Go Prometheus interceptors shows: buckets are tuned to service SLOs and truncated so that grpc_method and grpc_service are the only labels. Avoid adding user IDs or per-session tokens to stream labels.

Sampling is another pressure point. Logging every message on a stream that processes thousands of messages per second will overwhelm the log pipeline and distort latency measurements. Log stream lifecycle events at start and close, and sample per-message records at a fixed ratio with the trace ID attached. Streaming is where the structured logging patterns covered in our Go slog guide pay off: a limited set of high-value events, queryable by trace ID, is more effective than an unbounded log dump.

Fixing the Behavior Before the Metrics: Backpressure and Goroutine Shape

Instrumentation identifies the problem; correct stream code prevents it. The most common issue is unbounded buffering. The pattern that causes out-of-memory under a slow consumer is appending every incoming message to a slice in a private goroutine, then draining it later. The gRPC runtime already provides natural backpressure: Send() blocks when the peer’s receive buffer is full under HTTP/2 flow control (see the Rajpoot write-up of Go bidi streaming patterns). Allowing Send() to block is the correct design. Appending without a limit bypasses this and will cause the process to run out of memory.

The standard handler pattern uses two goroutines coordinated by an errgroup: one reads from the stream into a small buffered channel, the other drains that channel and sends replies. A bounded channel (a capacity near 16 is a common starting point) provides controlled backpressure between the reader and writer instead of an unbounded queue. Both goroutines must select on the stream context so that a client disconnect terminates them and returns the error instead of leaking a goroutine.

Reconnect storms deserve their own alert. Each reconnect reopens the HTTP/2 connection and re-registers the stream, so a client stuck in a tight retry loop can generate more load than the server can handle. Monitor client-side disconnect counters, not just aggregate message throughput. The stream that “flaps” often causes capacity issues.

Channel and Stream Sizing: When One Connection Is Not Enough

A single channel multiplexes onto a limited number of HTTP/2 connections, and each connection limits concurrent streams. When active streams reach that limit, new callers queue in the client and wait. This is the documented failure mode in the performance guide: for applications with high load or long-lived streams, the solution is either a separate channel per high-load area or a pool of channels with distinct arguments so the client does not reuse a single connection.

The production guidelines from that documentation are worth repeating. Reuse stubs and channels when possible. Use streaming only when it benefits the application, because a stream carrying work better suited to a bounded queue can reduce overall scalability. Treat channel count as a tunable: two bidi services that each want thousands of concurrent streams on one channel cause teams to hit the stream limit and then misattribute it to a network fault.

Keepalive Is Not Optional: Idle Connection Death Behind Proxies

Long-lived streams fail in a way unary calls rarely do: the connection silently drops mid-conversation when a load balancer or NAT closes an idle HTTP/2 connection. The gRPC keepalive server default sends a ping only after two hours, and the client default is effectively disabled, as the keepalive guidance states explicitly. On a long-lived bidi stream, no traffic means no detection of a dead peer.

The solution is deliberate keepalive configuration on both sides: a client interval well below the proxy idle timeout and a server enforcement policy that allows the client’s cadence. The caution is symmetric and important. Keepalive intervals below roughly one minute per connection are discouraged because frequent pings can resemble a denial-of-service attack, and a server that rejects the cadence responds with a GOAWAY whose debug string encodes too_many_pings. Set MinTime on the server high enough to accept your legitimate clients and load balancer, but no higher.

Keepalive adds a control frame at intervals, so it carries a small traffic cost. That cost is the price of not discovering a dead stream only when a user reports it. For the balance between observability cost and infrastructure budget, our microservices communication update explains how monitoring overhead can erase protocol gains when applied without care.

Choosing the Instrumentation Layer: Prometheus Interceptors vs OpenTelemetry

The table summarizes what the two sources above establish. There is no single correct choice. Prometheus interceptors keep all metrics in one Prometheus-compatible registry and provide the message counters a bidi stream requires, but they do not produce spans. OpenTelemetry offers trace correlation and a vendor-neutral export path (explained in the gRPC reference), but its per-stream value relies on the call-duration histogram, which is not granular enough to diagnose a slowly draining stream leg.

Many Go teams use both: one for RED metrics on a /metrics endpoint and one for trace spans linking a stream to its downstream calls. That setup is reasonable, though heavier than either alone. Start with the Prometheus path for bidi streams because of the message counters, then add OpenTelemetry tracing only to the streams important enough to justify running the exporter alongside.

Troubleshooting Checklist for Degraded Bidi Streams

When a bidi stream slows down, follow this diagnosis order. Each step is inexpensive and rules out a whole class of causes before changing code.

First, check message rate, not request rate. If grpc_server_msg_sent_total and msg_received_total are both flat while the producer is active, the peer is not draining; confirm with a consumer-side latency histogram. Second, look for reconnect flapping in the client disconnect counter; a tight retry loop generates volume without progress. Third, confirm keepalive is configured on both client and server and that the server’s MinTime accepts the client cadence; a missing GOAWAY with too_many_pings is a sign. Fourth, review goroutine structure and whether any private buffer can grow without limit; the fix is to let Send() apply backpressure instead of accumulating.

Finally, check whether the stream count on the channel is near the concurrent-stream limit. If callers are queuing at the client, the metric history will show stream starts flattening even as inbound demand rises. That indicates channel architecture (more channels or a pool), not the network layer you likely suspected first.

Key Takeaways:

• Flow control applies only to streaming RPCs and is enabled by default, so bidi degradation appears as resource pressure, not errors.
• Track message-level counters (grpc_server_msg_sent_total / msg_received_total), not request counts, to verify whether a stream is actually draining.
• Allow gRPC Send() to block for backpressure instead of buffering messages in an unbounded goroutine, which causes out-of-memory under slow consumers.
• Streams cannot be load balanced after starting, so channel and pool sizing affect scalability as much as application code.
• Keepalive is essential for long-lived streams: idle connections drop behind load balancers, and keepalive intervals below about one minute can trigger a too_many_pings GOAWAY.
• OpenTelemetry replaced OpenCensus for gRPC observability; Prometheus interceptors provide the message counters a bidi stream requires.

Sources and References

Sources cited while researching and writing this article:

• Flow Control – gRPC
• Performance Best Practices | gRPC
• OpenTelemetry Metrics | gRPC
• How to Monitor gRPC Services with Prometheus and Grafana
• Monitoring Golang gRPC Services with Prometheus Interceptors
• Keepalive | gRPC