Hi all, I found this exchange very helpful in clarifying the requirements around “multiplexing.”
It seems there are two different notions being discussed under the same term: 1. Transport-level multiplexing, where independent streams make progress independently (as with QUIC), primarily to avoid head-of-line blocking. 2. *Application-level interleaving*, where multiple logical data sources are serialized into a single, totally ordered stream with explicit sequencing semantics. Antoine’s point makes a lot of sense for (1): if the goal is parallelism and independence, this clearly belongs at the transport layer, and QUIC is a good fit. Rusty’s example, however, appears closer to (2): a single, ordered IPC stream where messages from different logical producers are intentionally interleaved, and later messages may depend on earlier ones across those logical streams (even with different schemas). In that case, preserving a single global ordering seems essential, and transport-level multiplexing alone doesn’t provide that semantic. >From that perspective, it feels reasonable to keep IPC itself simple and ordered and treat any “multiplexing” as an explicit, higher-level convention encoded within the IPC stream (e.g., via message types or annotations), rather than relying on transport behavior. This also suggests that it may be useful to scope IPC changes narrowly (e.g., ApplicationData / Empty messages with clear ordering semantics) and leave broader multiplexing or parallelism concerns to Flight or transport-layer choices. Thanks for the thoughtful discussion; it’s been very illuminating to follow. Best regards, Vignesh On Wed, 4 Feb 2026 at 22:35, Rusty Conover <[email protected]> wrote: > Hi Antoine, > > > Ideally IPC stream multiplexing should be done at the transport level, > > for example using QUIC instead of TCP. > > I hoped QUIC would give me what I’m after, but I don’t think it does. > > My (admittedly non-expert) understanding is that QUIC streams are > independently ordered: data on stream 2 is completely independent of data > on stream 1, and either can make progress at any time. That’s great for > avoiding head-of-line blocking, but it’s not the ordering model I desire. > > What I’m aiming for with interleaved/multiplexed IPC streams is a single, > explicit ordering across record batches from different logical streams. > That ordering allows data from IPC stream 2 to refer to previously > processed data from IPC stream 1, even though the two streams may have > different Arrow schemas. > > Because QUIC deliberately avoids imposing cross-stream ordering, I don’t > think it provides the semantics I’d want in this case. > > Best, > > Rusty >
