Part 2 of gRPC Under Pressure | An experiment in pending-work admission, transient latency, and the waiting we do not see.
Executive summary
Bounding a gRPC server’s application queue prevents an unlimited backlog, but it does not tell us how much pending work to admit. We compared three budgets inside one client-streaming RPC: fixed-32, fixed-8, and a feedback-driven budget between 1 and 32. Across five paired blocks, adaptive reduced the prespecified pooled server queue-residence p95 to 164.863 ms, versus 330.739 ms for fixed-8 and 1,321.235 ms for fixed-32. Yet during the slowdown alone, adaptive’s p95 was 657.598 ms, almost twice fixed-8’s 332.431 ms; the adverse tail appeared in every block. Adaptive also skipped more scheduled sends than fixed-32, while its recovery pattern was promising but partly censored. We did not measure per-message end-to-end latency. The lesson is not that feedback is useless: it is that a shorter server queue can shift waiting upstream, and adaptation must justify its complexity against a well-chosen fixed bound.
1. We bounded the queue. Now what?
In Part 1, we found an uncomfortable gap between gRPC transport flow control and application processing. HTTP/2 can limit how many DATA bytes cross a connection, yet a Rust handler can keep receiving decoded protobuf messages into an unbounded Tokio channel while a slower worker falls behind. Reserving bounded queue capacity before the next receive closed that particular hole.
But a bounded queue introduces another question: how large should the bound be? Thirty-two pending messages can absorb a burst, but each may wait behind the others. Eight may shorten that line, but may make the client wait sooner. If the worker slows down unexpectedly, perhaps the server should adjust the bound itself.
We built that adaptive policy, compared it with two fixed budgets, and froze the comparison before collecting the primary results. The outcome was less tidy than “adaptive wins”: it cut our primary server queue-delay metric sharply, yet some slowdown messages waited almost twice as long as under fixed-8. And a shorter server queue did not necessarily mean less waiting across the system.
This is a story about admission of pending messages inside one streaming RPC. We did not vary the number of concurrent RPCs, workers, or HTTP/2 streams. It is not a benchmark of adaptive RPC concurrency.
The question and the competing explanations
Research question: Can feedback-driven admission improve server-side queue residence and recovery without materially sacrificing throughput, and does it offer a meaningful advantage over a carefully chosen fixed budget?
We had three competing expectations. A feedback controller might follow changing worker capacity better than a fixed budget. A conservative fixed-8 budget might get most of the benefit without a control loop. And either approach might reduce time inside the queue simply by making work wait before admission. Those alternatives shaped the experiment before we saw the results.
What counted as success before we ran it?
- Primary queue target: at least 20% lower pooled enqueue-to-dequeue p95 than fixed-32 for messages enqueued in seconds 12–36.
- Throughput: at least 95% of fixed-32 worker completions per second over the 36-second offered interval, without materially more worker idle time while backlogged. The contract did not specify a numeric idle-materiality margin.
- Recovery: after service returns to normal at 24 seconds, reach a qualifying three-second window with queue p95 no worse than that run’s own healthy baseline and completion rate at least 95% of its healthy baseline. Whole-RPC duration must stay within 110% of fixed-32. The frozen confidence rule requires uncensored paired recovery observations.
- Added value: demonstrate a meaningful overall trade-off against fixed-8, not merely a win against fixed-32. No single weighted score or fixed-8 superiority margin was prespecified.
These were separate hypotheses, not one pass/fail badge. A primary p95 win could coexist with a weaker tail, shifted client waiting, or inconclusive recovery.
2. The small system we actually tested
Picture a telemetry sender feeding events to a service that performs one piece of work per event. Our Go load generator opens a single client-streaming Ingest(stream Payload) -> Summary RPC. The Rust server uses tonic and Tokio: one handler receives messages, a channel holds pending work, and one serial worker processes it. There is no database, external API, or per-message spawned task. The worker sleeps to simulate service time, so we can impose a repeatable slowdown without inventing downstream behavior.
| Go load generator | gRPC / HTTP/2 | Rust tonic handler | Tokio pending queue | One worker |
|---|---|---|---|---|
| Schedules up to 80 sends/s | Transports protobuf DATA | Reserves budget and channel slot before receiving | Physical K = 32; logical B varies | 10 → 40 → 10 ms service |
Conceptual architecture, not a measured trace. Transport credit, admission permits, channel slots, and completed processing are different resources.
Every message carries 16,384 pseudorandom payload bytes. The Go client schedules 80 opportunities per second for 36 seconds: 2,880 scheduled slots. It never catches up missed slots. If an earlier Send occupies the client when another scheduled slot arrives, that slot is recorded as skipped, not silently buffered for later. A successful Send is not an acknowledgment that the server processed the message; the RPC returns one final Summary after normal completion.
The service schedule is deliberately simple: 10 ms per message for seconds 0–12, 40 ms for seconds 12–24, and 10 ms again for seconds 24–36. The delay is selected when worker service starts. At 40 ms, a single worker can theoretically finish about 25 messages/s, well below the scheduled 80/s. The system must decide where the excess demand waits, or which send opportunities never occur.
Three layers matter here. HTTP/2 flow control accounts for DATA-byte credit between peers. Physical channel capacity limits actual pending messages in the Tokio channel. Logical admission budget determines whether this handler is allowed to reserve space for another message. None of these makes the worker faster.
3. Three policies, one physical queue
We kept the physical Tokio channel capacity at 32 for all policies. The only policy difference was the maximum pending-work budget, B.
| Policy | Physical queue K | Logical pending budget B | What changes? |
|---|---|---|---|
| Fixed-32 | 32 | 32 | Admit up to 32 pending messages or reservations |
| Fixed-8 | 32 | 8 | Admit up to 8 pending messages or reservations |
| Adaptive | 32 | Initially 16; range 1–32 | Change B using measured worker service time |
Transport bytes → handler admission gate → Tokio channel K = 32 → one worker
The same physical channel; three different limits on Q + R (queued messages + reserved slots):
Fixed-32: B = 32
Fixed-8: B = 8
Adaptive: B = 1–32 (starts at 16)
Bar shows the initial budget, not a measured trajectory.
HTTP/2 flow-control credit is independent. The executing worker is not counted in Q + R.Conceptual diagram. The logical budget restricts pending admissions without changing the physical queue capacity, HTTP/2 windows, or worker count.
Pending occupancy is Q + R: Q messages already enqueued, plus R reservations acquired but not yet handed off. The currently executing worker message, W, is not counted in B. All three policies use the same reservation gate and instrumentation. The handler must acquire a logical reservation and a physical slot before asking tonic for the next message.
The measured server’s critical ordering is short enough to show directly:
loop {
let reservation = reserve(shared.clone()).await?;
let next = tokio::select! { result=stream.message()=>result?,_=stopped.changed()=>return Err(Status::cancelled("worker terminated during receive")) };
if let Some(payload) = next {
reservation.handoff(payload)?;
} else {
drop(reservation);
Exact excerpt from server/mod.rs, accepted Stage 4 source. The excerpt ends inside the EOF branch; the source handles EOF, cancellation, accounting, and worker draining beyond these lines.
That order is the central safety mechanism. When the budget is exhausted, the handler waits before the next stream.message(). This may eventually slow the upstream sender through library buffering and HTTP/2 flow control, but it does not prove that the transport has zero internal buffering. If adaptive shrinks B below the occupancy already present, existing work is not discarded or retroactively revoked; new reservations stop until Q + R drops below B.
4. Can a feedback loop choose the right budget?
The controller is easier to understand as a feedback loop than as Rust code. It watches how long the worker actually takes, estimates how much waiting the current queue implies, and changes how many new messages the handler may admit. It cannot speed up the worker or revoke messages already accepted.
1. Observe
Every 500 ms: completed service durations and current Q
↓
2. Estimate
EWMA service time × Q; not a measured p95
↓
3. Decide
Two eligible high ticks: halve B; two eligible low and near-full ticks: B + 1; otherwise hold
↓
4. Admit
Next receive only when Q + R < B; worker keeps processing
↺ New worker completions feed the next observationConceptual feedback loop. A budget change controls future reservations inside one RPC. It is not adaptive RPC concurrency.
Our controller tries to keep an estimated queue delay near 100 ms. Every 500 ms, it looks at actual worker service durations completed during the interval. With fewer than five completions, it holds the budget and resets threshold streaks. Otherwise, it updates an exponentially weighted service-time estimate, giving 20% weight to the newest interval and 80% to the previous estimate.
It then computes a deliberately rough proxy:
Estimated queue delay = current queue depth Q × smoothed service time.
This is not the measured p95 and does not predict every message’s wait. It is a simple signal for the admission gate. Above 125 ms for two eligible ticks, the controller halves B, rounding up. Below 75 ms for two eligible ticks, while pending occupancy is near B, it adds one slot. B stays between 1 and 32. A decrease takes precedence; there is no special recovery override.
The actual frozen controller contains the decision logic:
let delay = q as f64 * ewma;
d.delay_ms = Some(delay);
self.high = if delay > 125.0 { self.high + 1 } else { 0 };
self.low = if delay < 75.0 && d.near_full {
self.low + 1
} else {
0
};
d.reason = "hold";
if self.high >= 2 {
d.proposed = b.div_ceil(2).max(1);
d.reason = "decrease";
self.high = 0;
self.low = 0;
} else if self.low >= 2 {
d.proposed = (b + 1).min(32);
d.reason = "increase";
self.high = 0;
self.low = 0;
}
Exact excerpt from controller.rs, accepted Stage 4 source. Eligibility, EWMA initialization, and the near-full predicate are defined in the surrounding source and frozen contract.
Why not simply choose a small fixed number? Because a larger budget can absorb short bursts, and a smaller one can avoid a long line when service slows. The adaptive hypothesis was that changing the budget could improve queue residence and recovery without materially sacrificing throughput. The counter-hypothesis was that adaptation would mostly relocate waiting upstream, while a conservative fixed budget might do nearly as well.
5. The result that initially looked like a win
We predeclared the primary endpoint as server queue-residence p95 for messages enqueued during seconds 12–36, covering slowdown and recovery. Queue residence begins when a message is enqueued and ends when the worker dequeues it. It does not include the time before admission, the client’s Send wait, or worker service.
| Policy | Pooled queue p95, 12–36 s | Pooled queue p99 | Enqueued messages in cohort |
|---|---|---|---|
| Fixed-32 | 1,321.235 ms | 1,325.096 ms | 6,439 |
| Fixed-8 | 330.739 ms | 332.546 ms | 6,312 |
| Adaptive | 164.863 ms | 658.698 ms | 6,302 |
Adaptive reduced the predeclared pooled p95 by 87.52% versus fixed-32, exceeding the frozen 20% target. It also reduced pooled p95 by 50.15% versus fixed-8. Those are real, favorable results for this precisely defined server-side metric.
We ran five fresh-process paired blocks, each containing all three policies in a prespecified seeded order: 15 valid runs in the corrected primary matrix. The quantiles above pool messages within each policy, rather than averaging five p95s. For uncertainty, the analysis resamples whole paired blocks, not individual messages. Its 95% block-bootstrap interval for the fixed-32 reduction is 87.48–87.56%, but with only five blocks that is descriptive repeatability under this setup, not a population guarantee.
The unequal cohort sizes are also important. Policies do not admit exactly the same messages during slowdown and recovery. A faster percentile in one admitted cohort is not proof that identical offered work completed faster under that policy.
6. The result that changed our interpretation
The pooled p95 is not the whole latency story. Split the measurement by phase, and the trade-off becomes obvious.
Pooled p95: enqueue at 12–36 s (slowdown + recovery; lower is better)
Fixed-32 · 1,321.235 ms
Fixed-8 · 330.739 ms
Adaptive · 164.863 ms
Slowdown-only p95: enqueue at 12–24 s (lower is better)
Fixed-32 · 1,324.535 ms
Fixed-8 · 332.431 ms
Adaptive · 657.598 ms
Reversal: adaptive beats fixed-8 on pooled p95, but fixed-8 beats adaptive on slowdown p95. Bars use a common zero baseline and the same 1,324.535 ms maximum across both panels.Measured comparison, Stage 5. Pooled nearest-rank queue-residence p95 from five runs per policy. These are different enqueue-time cohorts and have unequal admitted-message counts; the bars are not end-to-end latency.
| Queue residence for messages enqueued during slowdown, 12–24 s | Fixed-32 | Fixed-8 | Adaptive |
|---|---|---|---|
| Pooled p95 | 1,324.535 ms | 332.431 ms | 657.598 ms |
| Pooled p99 | 1,330.429 ms | 333.658 ms | 661.579 ms |
Against fixed-8, adaptive’s slowdown p95 was almost 1.98 times as large. The worse slowdown p95 and p99 persisted in all five paired blocks. The median of per-run slowdown p95 values was 658.998 ms adaptive versus 332.557 ms fixed-8. This is not a single anomalous run that can be dismissed.
Why can adaptive win pooled p95 while losing this slowdown tail? The pooled window includes recovery messages, many of which experience almost no queue residence. The slowdown-only distribution isolates the difficult period. Adaptive’s pooled p99, at 658.698 ms, also remains worse than fixed-8’s 332.546 ms. The apparent win depends on which part of which cohort we ask about.
The controller trajectory provides a plausible mechanism. It begins with B=16, allowing a larger transient backlog than fixed-8 before feedback has time to shrink the budget. In four runs, the budget steps 16 → 8 at about 13 seconds, then 4, then 2, and later 3; one run delays the later steps. The controller cannot make already queued messages disappear. This is consistent with the observed tail penalty, but we should not pretend to have causally attributed each slow message to a particular tick.
This was the moment the engineering question changed. We were no longer asking only whether the controller could make the queue look shorter. We had to ask what it cost, and where waiting went.
7. The queue got shorter. Where did the waiting go?
Measurement boundary first: enqueue-to-dequeue time is a local server metric. Logical admission wait occurs before enqueue. A Go Send wall duration measures the client’s call, not server completion or a separately identified HTTP/2 stall. These timers cover different events and populations, so we cannot add their summaries together to manufacture an end-to-end latency figure.
A queue-residence timer starts after admission. If the handler is waiting for a logical reservation before it reads the next message, that wait is invisible to the queue-residence histogram. Transport and library buffers may absorb some work; eventually a blocked or delayed Go Send can prevent future scheduled opportunities from being attempted. Moving the waiting point can improve a local metric without improving the sender’s experience.
| Before server admission | Inside server queue | Worker |
|---|---|---|
| Scheduled Go slot → Send call → grpc-go / HTTP/2 → handler waits for reservation | Enqueue → dequeue Measured queue residence | Service start → completion |
Conceptual timing boundary. The Stage 5 queue p95 covers only enqueue-to-dequeue. The experiment does not reconstruct a per-message end-to-end latency across client and server clocks.
All three policies scheduled 2,880 send slots per run. Their median skipped slots were 630 for fixed-32, 655 for fixed-8, and 660 for adaptive. Adaptive skipped more than fixed-32 in all five paired blocks, by a mean of 27.4 slots. Since skipped opportunities were never retried, those are different amounts of offered work that actually entered the pipeline.
The client-side Send measurements are consistent with upstream waiting, but they are not a packet-level measurement of HTTP/2 stalls. Adaptive’s longest Send in each run was 359–370 ms, versus at most 46 ms for either fixed policy. The longest adaptive calls began near the first budget shrink. That timing is suggestive, not an isolated causal proof of which transport buffer or flow-control window blocked.
Worker throughput was close across policies: median completions during the 36-second interval were 62.500/s fixed-32, 61.806/s fixed-8, and 61.667/s adaptive. Adaptive reached 98.799% of fixed-32’s paired mean throughput, above the frozen 95% threshold. But the full throughput-and-idle hypothesis was not established: the adaptive worker’s idle-with-backlog fraction was higher by 0.158 percentage points on average, and the contract did not define a numerical materiality margin. The absolute difference was small, roughly 95 ms versus 90 ms of idle-with-backlog time, so neither “serious starvation” nor “proven noninferiority” is justified.
Whole-RPC durations clustered near 36.35 seconds, but this is not an equal-work completion benchmark. The client follows a 36-second schedule and does not make up skipped slots. Successful Send calls are not individual processing acknowledgments. We did not measure an end-to-end per-message latency improvement, and we cannot infer one from queue residence or similar RPC durations.
8. Recovery looked promising, but the evidence has boundaries
When the worker returns to 10 ms service at second 24, can the queue return to its healthy operating band? We froze a recovery criterion before collection: a qualifying continuous three-second window whose queue-residence p95 is at or below that run’s own healthy baseline and whose worker completion rate is at least 95% of its healthy baseline. The table reports confirmation delay after second 24, including the three-second qualifying window.
| Paired block | Fixed-32 | Fixed-8 | Adaptive |
|---|---|---|---|
| 1 | 7.2 s | 7.6 s | 4.2 s |
| 2 | >12 s, censored | >12 s, censored | 7.4 s |
| 3 | 7.2 s | 4.8 s | 5.6 s |
| 4 | >12 s, censored | 7.8 s | 4.2 s |
| 5 | 7.4 s | 7.2 s | 4.2 s |
Adaptive met the operational criterion in 5/5 runs, fixed-8 in 4/5, and fixed-32 in 3/5. That pattern favors adaptive. But a >12 s entry means the criterion was not confirmed within the observation window, not that recovery occurred at 12 seconds. The preregistered complete-pair confidence rule cannot produce an unconditional recovery advantage with censored fixed-32 baselines. And fixed-8 recovered sooner than adaptive in block 3. The policy-specific healthy thresholds also vary with small scheduling outliers, making this definition sensitive to baseline noise.
There was another uncomfortable finding, this time in the harness. During the original collection, a marker-file race allowed the server to read an empty startup marker; the original fixed-8 attempt never began its workload and hit the unchanged deadline. We preserved the two valid original attempts and the invalid one, documented the diagnosis, and changed only the harness to atomically publish the marker. Before restarting, we recorded the amendment and kept the application, controller, workload, binaries, order, and analysis definitions unchanged. Then we restarted the entire 15-run matrix, not just the failed policy. The audit trail contains 18 total attempts: 17 valid measurements and one invalid startup; the corrected primary cohort contains 15 valid runs. That is an explicit departure from the original no-replacement plan, not a hidden cherry-pick.
Finally, this is a small, controlled experiment: one host, loopback plaintext transport, one stream, one worker, one workload and seed, five paired blocks, Rust debug build, and a fixed slowdown schedule. The block order was seeded and interleaved, but not perfectly position-balanced. The block-bootstrap intervals are informative within these conditions; they do not establish behavior under real networks, many concurrent RPCs, downstream failures, or production traffic distributions.
9. Was the adaptive controller worth it?
It depends on the question. If the goal is to reduce the predeclared pooled server queue-residence p95 relative to fixed-32, the adaptive policy succeeded in this workload. If the goal is to outperform a sensible small fixed budget across slowdown tails, upstream waiting, throughput, and recovery, we did not establish that. Fixed-8 delivered a much better slowdown p95/p99 than adaptive, with similar throughput and whole-RPC duration. Adaptive showed favorable recovery observations, but the frozen inference rule and censoring prevent an unconditional recovery claim.
When should you start with a fixed budget? When worker capacity is reasonably stable, the overload envelope is understood, and a small bound meets queue-tail and throughput objectives, fixed admission is easier to operate and reason about. In this experiment fixed-8 delivered a much shorter slowdown tail than adaptive, with nearly the same worker throughput. That makes it a credible first implementation, not a fallback to be embarrassed about.
When might adaptation earn its complexity? When downstream capacity changes enough that no single fixed budget meets both burst absorption and tail-latency objectives, and repeated workload tests demonstrate a benefit against a tuned fixed baseline. A controller also needs operational safeguards: visible budgets and occupancy, hysteresis, cancellation-safe permits, and tests for shrink-with-backlog. Our single-stream experiment did not establish that production case.
What should you measure? Track scheduled, attempted, completed, and skipped client sends; admission-gate wait; physical queue occupancy and queue residence by phase; worker service and completions; worker idle-with-backlog; client Send distributions and maxima; controller budget changes; and real end-to-end per-message latency if that is the product objective. Match offered workloads and explicitly account for messages that never enter the system.
What remains untested? Multiple streams and workers, real networks, release-mode performance, diverse service-time distributions, sustained or repeated overload, and downstream processing acknowledgments. This study supports a local queue-residence improvement and a repeatable transient-tail cost, not a universal ranking of adaptive and fixed admission.
A shorter queue is not necessarily a faster system. Admission control determines where work is allowed to wait; it does not create processing capacity. That is the engineering decision this experiment made visible.
Reproduce and inspect the evidence: Frozen EXPERIMENT-002 contract · Stage 5 report · primary analysis JSON · per-run CSV · exclusions ledger · verification JSON.
gRPC Under Pressure
A three-part engineering series on backpressure, admission control, and overload.