+1, this builds on RTM's community-reviewed shuffle infrastructure instead of 
having to deal with introducing the extra complexity of adding that to Spark 
entirely separately.

On 2026-11-12 01:34 UTC cheng xu via dev wrote:
> Hi Spark community,
> 
> I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM)
> execution infrastructure to eligible distributed batch SQL and DataFrame
> queries.
> 
> Agentic applications increasingly use SQL for retrieval, feature
> generation, ranking, and analytical tool calls. For these interactive
> workloads, materialized shuffle boundaries can add latency by requiring
> downstream stages to wait for upstream shuffle output. The proposal allows
> downstream stages to consume intermediate data as it arrives, overlapping
> producer computation, cross-executor transfer, and consumer computation
> while avoiding unnecessary shuffle materialization.
> 
> SPIP document:
> https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le
> 
> 
> JIRA: https://issues.apache.org/jira/browse/SPARK-61454
> 
> The proposed approach builds on RTM’s existing connected-stage execution
> and distributed transport, with three main additions:
> 
>    - Planner selection of eligible batch shuffle boundaries, with explicit
>    opt-in and materialized execution retained for unsupported plans and
>    exchanges.
>    - Consumer-first admission with resident reader capacity, bounded
>    producer concurrency, and aggregate buffer budgets to preserve progress
>    under bounded resources.
>    - Batch-specific lifecycle handling to isolate execution attempts and
>    release transport and scheduler state on completion, cancellation, or
>    failure.
> 
> The initial scope targets shorter, read-only queries for which restarting
> the whole query after a transient failure is acceptable. Materialized
> shuffle remains the default. AQE integration and automatic group retry are
> follow-up work, not part of the initial milestone.
> 
> Early prototype results include a 1.64x speedup on distributed TPC-H
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%.
> These are workload-specific results, not a general performance claim. The
> broader goal is to reduce p95 agent-step latency and increase completed
> steps per minute on a fixed cluster; concurrent-session validation is part
> of the proposed qualification work.
> 
> I’d particularly appreciate feedback on the initial scope, the
> reader-reservation and producer-admission model, memory bounds and failure
> handling, and the proposed qualification criteria. I’d also welcome a PMC
> member interested in shepherding this SPIP.
> 
> Please share feedback on the JIRA or in this thread.
> 

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to