Skip to content

proto: add runtime set_stream_receive_window and FlowControlStats#2582

Open
JavaPerformance wants to merge 3 commits intoquinn-rs:mainfrom
JavaPerformance:runtime-stream-window-and-flow-control-stats
Open

proto: add runtime set_stream_receive_window and FlowControlStats#2582
JavaPerformance wants to merge 3 commits intoquinn-rs:mainfrom
JavaPerformance:runtime-stream-window-and-flow-control-stats

Conversation

@JavaPerformance
Copy link
Copy Markdown

Summary

Two related additions:

1. Connection::set_stream_receive_window(VarInt)

Runtime adjustment of the per-stream receive window. This completes the runtime window adjustment API — set_receive_window() (connection-level) and set_send_window() already exist.

2. FlowControlStats on ConnectionStats

pub struct FlowControlStats {
    pub send_credit_remaining: u64,
    pub recv_credit_remaining: u64,
    pub streams_blocked_by_conn_fc: u64,
}

Motivation

Stream window setter

On high-BDP paths (e.g., 250ms RTT cross-region), the default stream receive window can become a throughput bottleneck. Applications that monitor path characteristics and adapt windows need to adjust both connection-level and stream-level windows at runtime. Connection-level is already possible; stream-level is not.

The implementation is minimal because stream_receive_window is stored as a single field on StreamsState and passed by value to max_stream_data() on each flow control decision. Changing it immediately affects the next MAX_STREAM_DATA frame for any stream without iterating per-stream state.

Only expansion is safe: MAX_STREAM_DATA is a one-way ratchet per RFC 9000 — previously advertised limits cannot be revoked. The doc comment makes this clear.

Flow control stats

FrameStats already exposes data_blocked and stream_data_blocked frame counts, which tell you how many times the peer was blocked. FlowControlStats complements this with real-time utilization:

  • send_credit_remaining — how much headroom before we hit the peer's MAX_DATA
  • recv_credit_remaining — how much headroom before the peer exhausts our MAX_DATA
  • streams_blocked_by_conn_fc — number of streams currently waiting on connection-level flow control

This enables applications to detect flow control pressure before it becomes a bottleneck, rather than only after DATA_BLOCKED frames have been sent.

Changes

  • quinn-proto/src/connection/streams/state.rsFlowControlStats struct, set_stream_receive_window(), flow_control_stats()
  • quinn-proto/src/connection/streams/mod.rs — re-export
  • quinn-proto/src/connection/mod.rs — expose on Connection, populate in stats()
  • quinn-proto/src/connection/stats.rs — add flow_control field to ConnectionStats
  • quinn-proto/src/lib.rs — crate-level re-export
  • quinn/src/connection.rs — public set_stream_receive_window()

All new types use #[non_exhaustive]. Backwards-compatible.

Related

Add Connection::set_stream_receive_window() for runtime adjustment of
the per-stream receive window. This completes the runtime window
adjustment API — connection-level set_receive_window() and
set_send_window() already exist.

The implementation is minimal because stream_receive_window is stored
as a single field on StreamsState and passed by value to
max_stream_data() on each flow control decision. Changing it
immediately affects the next MAX_STREAM_DATA frame for any stream
without iterating per-stream state.

Only expansion is safe at the QUIC protocol level: MAX_STREAM_DATA is
a one-way ratchet and previously advertised limits cannot be revoked.

Also adds FlowControlStats to ConnectionStats with:
- send_credit_remaining: peer's MAX_DATA minus data sent
- recv_credit_remaining: our MAX_DATA minus data received
- streams_blocked_by_conn_fc: streams blocked on connection flow control

These complement the existing FrameStats.data_blocked and
stream_data_blocked counters (which count how many times the peer was
blocked) with real-time utilization snapshots.
Copy link
Copy Markdown
Collaborator

@Ralith Ralith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applications that monitor path characteristics and adapt windows need to adjust both connection-level and stream-level windows at runtime.

Why not configure whatever upper bound you're willing to tolerate up front?

///
/// This is the peer's advertised `MAX_DATA` limit minus the sum of data offsets across all
/// send streams. When this reaches zero, the connection is send-blocked at the flow control
/// level and a `DATA_BLOCKED` frame will be emitted.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and a DATA_BLOCKED frame will be emitted.

This is false. Quinn does not currently implement sending DATA_BLOCKED.

pub send_credit_remaining: u64,
/// Remaining receive credit on the connection
///
/// This is our advertised `MAX_DATA` limit minus the sum of data received across all receive
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/data/max offsets/ to match the precision above, perhaps?

/// Only expansion is safe at the QUIC protocol level. Shrinking does not revoke previously
/// advertised limits; it only reduces what is advertised on future updates.
pub(crate) fn set_stream_receive_window(&mut self, window: u64) {
self.stream_receive_window = window;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also trigger transmission of new MAX_STREAM_DATA frames where appropriate? If not, document why.

@JavaPerformance
Copy link
Copy Markdown
Author

Good question. Two reasons:

  1. Memory budget across many connections. A relay proxy may handle hundreds of connections to backends with very different path characteristics (LAN vs cross-continent). Setting the upper bound to what the worst-case WAN path needs (e.g., 32 MiB) for every connection wastes memory on
    LAN connections that only need 2 MiB. We start conservative and grow per-connection based on observed BDP, so the aggregate memory stays proportional to actual need rather than worst-case.
  2. The initial window advertisement affects the peer immediately. If we configure a 32 MiB stream window up front, the receiver advertises that full credit to the sender at connection start. On paths where the actual BDP is 2 MB, this over-advertises by 16x — the sender can have 32 MiB
    in flight before we can signal backpressure. Starting small and growing based on observed throughput keeps the advertised credit proportional to what the path can actually sustain.

set_receive_window already exists for the connection-level case and the same reasoning applies at the stream level.

On the doc comments:

  • Will fix the DATA_BLOCKED claim — thanks for the correction.
  • Will use "max offsets" for precision on the receive side.
  • On whether set_stream_receive_window should trigger MAX_STREAM_DATA transmission: it doesn't need to because max_stream_data() is called on every flow control evaluation and reads the current value of stream_receive_window at that point. The next time the receiver consumes data and
    Quinn checks whether to send a MAX_STREAM_DATA update, it will use the new (larger) window and advertise accordingly. No explicit trigger is needed — it takes effect on the next natural flow control cycle. Will add a note.

- Remove false claim that DATA_BLOCKED frame is emitted (Quinn does
  not currently implement sending DATA_BLOCKED)
- Use "max offsets" instead of "data received" for recv_credit precision
- Document why set_stream_receive_window does not trigger explicit
  MAX_STREAM_DATA transmission (max_stream_data() reads the current
  value on every flow control evaluation)
@Ralith
Copy link
Copy Markdown
Collaborator

Ralith commented Mar 22, 2026

it takes effect on the next natural flow control cycle

That was clear, but why is that necessarily okay? There's no guarantee that flow control for a specific stream will be refreshed on any particular time scale.

@Ralith
Copy link
Copy Markdown
Collaborator

Ralith commented Mar 22, 2026

Setting the upper bound to what the worst-case WAN path needs (e.g., 32 MiB) for every connection wastes memory on LAN connections that only need 2 MiB

Where is that memory wasted? These are upper bounds; we shouldn't actually commit the memory if we don't have data to store.

If you're worried about badly behaved peers forcing excess memory consumption with synthetic workloads, such peers could just as well spoof your heuristic by faking high latency.

@JavaPerformance
Copy link
Copy Markdown
Author

There's no guarantee that flow control for a specific stream will be refreshed on any particular time scale.

You're right — if a stream is idle, the new window won't be advertised until data flows again. In our use case this is acceptable because we only grow the window on connections that are actively transferring (we measure BDP from observed goodput). An idle stream doesn't need a larger window advertised.

That said, if the expectation is that set_stream_receive_window should take effect promptly for all open streams, it would need to iterate recv state and mark streams for MAX_STREAM_DATA retransmission — similar to how set_receive_window queues pending.max_data. Happy to add that if you think it's the right behavior. The implementation would be straightforward since the Recv instances don't store the window — they'd just need to be marked for a flow control update on their next check.

Where is that memory wasted? These are upper bounds; we shouldn't actually commit the memory if we don't have data to store.

Fair point — the receiver doesn't pre-allocate. The cost is on the sender side: a large advertised window lets the sender have more data in flight, which consumes sender-side send buffer memory. But you're right that this is the sender's problem, not the receiver's.

The stronger motivation is the second one: over-advertising credit on a low-BDP path means the sender can burst far more data than the path can sustain in one RTT. That creates large queues at intermediate routers and inflates latency. Starting conservative and growing based on measured path capacity keeps the advertised credit proportional to what the network can absorb without bufferbloat.

That said, for many applications, configuring the maximum tolerable window up front and letting the peer figure out pacing is perfectly fine. The runtime setter is most useful for long-lived connections (hours/days) where path characteristics change, or for relay proxies that handle heterogeneous paths and want to minimize aggregate in-flight data across many connections.

@Ralith
Copy link
Copy Markdown
Collaborator

Ralith commented Mar 22, 2026

An idle stream doesn't need a larger window advertised

I'm happy relying on this assumption for simplicity so long as we document it, so there's no uncertainty if we need to revisit it in the future.

The stronger motivation is the second one: ...

This sounds a lot like trying to implement congestion control at the application layer. Would it make more sense to use the congestion controller for this, and let senders set their own internal buffer sizes according to their preference?

@JavaPerformance
Copy link
Copy Markdown
Author

I'm happy relying on this assumption for simplicity so long as we document it

Will do — I'll add a doc note that the new window only takes effect on the next natural flow control update, and idle streams won't see it until they become active again.

This sounds a lot like trying to implement congestion control at the application layer.

Fair criticism — that argument was weaker than I made it sound. The congestion controller should handle pacing, you're right.

The simpler and honest motivation: set_receive_window already exists for the connection level. This adds the stream-level equivalent for API completeness. Our use case is long-lived relay connections where we want to grow stream windows after observing path characteristics, the same way we already grow connection windows at runtime. That's it.

I'll update the PR description to focus on API completeness rather than the bufferbloat rationale.

Document that idle streams won't see the updated window until they
become active again, per reviewer feedback.
Copy link
Copy Markdown
Collaborator

@Ralith Ralith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include tests that exercise the flow control update transmit path for both growing and shrinking the window. Of particular interest is the case where the window is reduced immediately after sending a flow control update.

/// value and advertise accordingly.
///
/// Only expansion is safe at the QUIC protocol level. Shrinking does not revoke previously
/// advertised limits; it only reduces what is advertised on future updates.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is a good description of what the behavior should be, but I'm not convinced it's accurate to what the behavior is as written. For example, if a flow control credit update is scheduled for a stream whose receive window has been shrunk, the new value will be computed based on the latest stream_receive_window value, when instead it should be the high water mark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants