+1, the performance results look promising Kent
________________________________ From: cheng xu via dev <[email protected]> Sent: Thursday, November 12, 2026 9:34 AM To: [email protected] <[email protected]> Cc: cheng xu <[email protected]> Subject: [DISCUSS] SPIP: Extend Real-Time Mode to Distributed Batch SQL and DataFrame Queries Hi Spark community, I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM) execution infrastructure to eligible distributed batch SQL and DataFrame queries. Agentic applications increasingly use SQL for retrieval, feature generation, ranking, and analytical tool calls. For these interactive workloads, materialized shuffle boundaries can add latency by requiring downstream stages to wait for upstream shuffle output. The proposal allows downstream stages to consume intermediate data as it arrives, overlapping producer computation, cross-executor transfer, and consumer computation while avoiding unnecessary shuffle materialization. SPIP document: https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le JIRA: https://issues.apache.org/jira/browse/SPARK-61454 The proposed approach builds on RTM’s existing connected-stage execution and distributed transport, with three main additions: * Planner selection of eligible batch shuffle boundaries, with explicit opt-in and materialized execution retained for unsupported plans and exchanges. * Consumer-first admission with resident reader capacity, bounded producer concurrency, and aggregate buffer budgets to preserve progress under bounded resources. * Batch-specific lifecycle handling to isolate execution attempts and release transport and scheduler state on completion, cancellation, or failure. The initial scope targets shorter, read-only queries for which restarting the whole query after a transient failure is acceptable. Materialized shuffle remains the default. AQE integration and automatic group retry are follow-up work, not part of the initial milestone. Early prototype results include a 1.64x speedup on distributed TPC-H scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These are workload-specific results, not a general performance claim. The broader goal is to reduce p95 agent-step latency and increase completed steps per minute on a fixed cluster; concurrent-session validation is part of the proposed qualification work. I’d particularly appreciate feedback on the initial scope, the reader-reservation and producer-admission model, memory bounds and failure handling, and the proposed qualification criteria. I’d also welcome a PMC member interested in shepherding this SPIP. Please share feedback on the JIRA or in this thread.
