Hi Spark community,

I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM)
execution infrastructure to eligible distributed batch SQL and DataFrame
queries.

Agentic applications increasingly use SQL for retrieval, feature
generation, ranking, and analytical tool calls. For these interactive
workloads, materialized shuffle boundaries can add latency by requiring
downstream stages to wait for upstream shuffle output. The proposal allows
downstream stages to consume intermediate data as it arrives, overlapping
producer computation, cross-executor transfer, and consumer computation
while avoiding unnecessary shuffle materialization.

SPIP document:
https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le


JIRA: https://issues.apache.org/jira/browse/SPARK-61454

The proposed approach builds on RTM’s existing connected-stage execution
and distributed transport, with three main additions:

   - Planner selection of eligible batch shuffle boundaries, with explicit
   opt-in and materialized execution retained for unsupported plans and
   exchanges.
   - Consumer-first admission with resident reader capacity, bounded
   producer concurrency, and aggregate buffer budgets to preserve progress
   under bounded resources.
   - Batch-specific lifecycle handling to isolate execution attempts and
   release transport and scheduler state on completion, cancellation, or
   failure.

The initial scope targets shorter, read-only queries for which restarting
the whole query after a transient failure is acceptable. Materialized
shuffle remains the default. AQE integration and automatic group retry are
follow-up work, not part of the initial milestone.

Early prototype results include a 1.64x speedup on distributed TPC-H
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%.
These are workload-specific results, not a general performance claim. The
broader goal is to reduce p95 agent-step latency and increase completed
steps per minute on a fixed cluster; concurrent-session validation is part
of the proposed qualification work.

I’d particularly appreciate feedback on the initial scope, the
reader-reservation and producer-admission model, memory bounds and failure
handling, and the proposed qualification criteria. I’d also welcome a PMC
member interested in shepherding this SPIP.

Please share feedback on the JIRA or in this thread.

Reply via email to