Thanks Jerry for the question. IMO, for the blocking operator (e.g., aggregation), it will have a similar behavior as before. If it hits memory pressure, spill (within operator) will happen.
And the backpressure may help control the producer with a consideration of used memory footprint. On 2026/11/17 06:13:15 Jerry Peng wrote: > +1, I’m excited to see RTM extended to more use cases. I’d still like to > understand how you plan to handle blocking batch operators that consume all > their input before producing output, and whether making those operators > incremental is part of the proposal. > > On Wed, Nov 11, 2026 at 5:35 PM cheng xu via dev <[email protected]> > wrote: > > > Hi Spark community, > > > > I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM) > > execution infrastructure to eligible distributed batch SQL and DataFrame > > queries. > > > > Agentic applications increasingly use SQL for retrieval, feature > > generation, ranking, and analytical tool calls. For these interactive > > workloads, materialized shuffle boundaries can add latency by requiring > > downstream stages to wait for upstream shuffle output. The proposal allows > > downstream stages to consume intermediate data as it arrives, overlapping > > producer computation, cross-executor transfer, and consumer computation > > while avoiding unnecessary shuffle materialization. > > > > SPIP document: > > https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le > > > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-61454 > > > > The proposed approach builds on RTM’s existing connected-stage execution > > and distributed transport, with three main additions: > > > > - Planner selection of eligible batch shuffle boundaries, with > > explicit opt-in and materialized execution retained for unsupported plans > > and exchanges. > > - Consumer-first admission with resident reader capacity, bounded > > producer concurrency, and aggregate buffer budgets to preserve progress > > under bounded resources. > > - Batch-specific lifecycle handling to isolate execution attempts and > > release transport and scheduler state on completion, cancellation, or > > failure. > > > > The initial scope targets shorter, read-only queries for which restarting > > the whole query after a transient failure is acceptable. Materialized > > shuffle remains the default. AQE integration and automatic group retry are > > follow-up work, not part of the initial milestone. > > > > Early prototype results include a 1.64x speedup on distributed TPC-H > > scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. > > These are workload-specific results, not a general performance claim. The > > broader goal is to reduce p95 agent-step latency and increase completed > > steps per minute on a fixed cluster; concurrent-session validation is part > > of the proposed qualification work. > > > > I’d particularly appreciate feedback on the initial scope, the > > reader-reservation and producer-admission model, memory bounds and failure > > handling, and the proposed qualification criteria. I’d also welcome a PMC > > member interested in shepherding this SPIP. > > > > Please share feedback on the JIRA or in this thread. > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
