Thanks Jerry for the question. 

IMO, for the blocking operator (e.g., aggregation), it will have a similar 
behavior as before. If it hits memory pressure, spill (within operator) will 
happen.

And the backpressure may help control the producer with a consideration of used 
memory footprint. 

On 2026/11/17 06:13:15 Jerry Peng wrote:
> +1, I’m excited to see RTM extended to more use cases. I’d still like to
> understand how you plan to handle blocking batch operators that consume all
> their input before producing output, and whether making those operators
> incremental is part of the proposal.
> 
> On Wed, Nov 11, 2026 at 5:35 PM cheng xu via dev <[email protected]>
> wrote:
> 
> > Hi Spark community,
> >
> > I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM)
> > execution infrastructure to eligible distributed batch SQL and DataFrame
> > queries.
> >
> > Agentic applications increasingly use SQL for retrieval, feature
> > generation, ranking, and analytical tool calls. For these interactive
> > workloads, materialized shuffle boundaries can add latency by requiring
> > downstream stages to wait for upstream shuffle output. The proposal allows
> > downstream stages to consume intermediate data as it arrives, overlapping
> > producer computation, cross-executor transfer, and consumer computation
> > while avoiding unnecessary shuffle materialization.
> >
> > SPIP document:
> > https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le
> >
> >
> > JIRA: https://issues.apache.org/jira/browse/SPARK-61454
> >
> > The proposed approach builds on RTM’s existing connected-stage execution
> > and distributed transport, with three main additions:
> >
> >    - Planner selection of eligible batch shuffle boundaries, with
> >    explicit opt-in and materialized execution retained for unsupported plans
> >    and exchanges.
> >    - Consumer-first admission with resident reader capacity, bounded
> >    producer concurrency, and aggregate buffer budgets to preserve progress
> >    under bounded resources.
> >    - Batch-specific lifecycle handling to isolate execution attempts and
> >    release transport and scheduler state on completion, cancellation, or
> >    failure.
> >
> > The initial scope targets shorter, read-only queries for which restarting
> > the whole query after a transient failure is acceptable. Materialized
> > shuffle remains the default. AQE integration and automatic group retry are
> > follow-up work, not part of the initial milestone.
> >
> > Early prototype results include a 1.64x speedup on distributed TPC-H
> > scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%.
> > These are workload-specific results, not a general performance claim. The
> > broader goal is to reduce p95 agent-step latency and increase completed
> > steps per minute on a fixed cluster; concurrent-session validation is part
> > of the proposed qualification work.
> >
> > I’d particularly appreciate feedback on the initial scope, the
> > reader-reservation and producer-admission model, memory bounds and failure
> > handling, and the proposed qualification criteria. I’d also welcome a PMC
> > member interested in shepherding this SPIP.
> >
> > Please share feedback on the JIRA or in this thread.
> >
> 

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to