Or we could hold off until after the first community sync end of November if you wanted to bring it up there?
On Mon, Nov 16, 2026 at 2:36 PM Holden Karau via dev <[email protected]> wrote: > I think discussion on the doc has settled if you want to move it forward > to a VOTE Tornike? > > On 2026-10-25 13:28 UTC Tornike Gurgenidze wrote: > > Hi all, > > > > I'd like to start a discussion on a SPIP to add a built-in ADBC (Arrow > > Database Connectivity) data source to Spark, alongside the existing JDBC > > one. > > > > - SPIP doc: > > > https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY > > - JIRA: https://issues.apache.org/jira/browse/SPARK-60255 > > - Earlier discussion: https://github.com/apache/spark/issues/54603 > > - Prototype: https://github.com/tokoko/spark-adbc > > > > Motivation > > > > JDBC returns data one row at a time, and Spark reads each value with a > > separate getXxx call. Many analytical databases store and send data in > > columns. Reading them through JDBC means converting columns to rows in > the > > driver, reading each value one by one in Spark, and then converting back > to > > columns again if Comet or Gluten is used. ADBC drivers return Arrow > record > > batches instead. Spark can wrap those as ColumnarBatches through > > ArrowColumnVector, almost without copying. On the prototype, reads from > > columnar databases are 3-10x faster than JDBC, and the gap grows on wider > > tables. > > > > What's proposed > > > > - A new optional "adbc" format. There are no public API changes, and the > > JDBC source is unchanged. ADBC complements JDBC; it doesn't replace it. > > - Pushdown of column pruning, filters, limit, Top-N and aggregates > > (COUNT/SUM/MIN/MAX/AVG). > > - Two kinds of partitioning that can be combined: > > - JDBC-style range partitioning > > (partitionColumn/lowerBound/upperBound/numPartitions). > > - Driver partitioning through ADBC's executePartitioned, where each > > descriptor the driver returns becomes one Spark task. Flight SQL servers > > already split results this way. > > - Appending writes through ADBC bulk ingest, one transaction per task > where > > the driver supports it. This is the same at-least-once guarantee as the > > JDBC writer. > > - No per-database dialect classes in Spark. The driver reports the few > > syntax choices pushdown needs through GetInfo/SqlInfo, and filter values > > are sent as bound Arrow parameters, not literals. As a second path, > > Substrait plans can be handed to drivers that translate them to SQL > > themselves. > > > > Out of scope for the first version: atomic job-level writes and > Structured > > Streaming. > > > > Upstream dependencies. Two Arrow PRs would improve this. Neither blocks a > > first version: > > > > - apache/arrow#49796: SqlInfo codes for limit/offset syntax, NULLS > > ordering, LIKE escaping and parameter markers. > > - apache/arrow-adbc#4317: a partitioned bulk ingest API. It maps directly > > onto Spark's write commit protocol, which would make writes atomic across > > the whole job. > > > > The prototype currently runs a shared test suite against 13 engines: > > PostgreSQL, MySQL, MSSQL, DuckDB, ClickHouse, Trino, Presto, DataFusion, > > SingleStore, Exasol, Oracle, Flight SQL (GizmoSQL) and Spark Connect. > > > > Longer-term direction > > > > The first goal is narrow: a faster alternative to JDBC for databases that > > ship ADBC drivers. Longer term, I think ADBC could also become a general > > way for non-JVM sources to integrate with Spark. ADBC already covers > > catalog metadata (GetObjects, GetTableSchema), driver-planned partitioned > > reads, and bulk ingest, with partitioned ingest proposed in > > apache/arrow-adbc#4317. Substrait pushdown would let a driver accept more > > than SQL. Together, these could let catalogs and table formats with an > ADBC > > driver plug into Spark, for example through a TableCatalog backed by > > GetObjects, without a Spark-specific connector. To try the idea out, I've > > put together a small proof of concept: an ADBC driver for Iceberg built > on > > iceberg-rust (https://github.com/tokoko/adbc-iceberg). It's early and > > experimental so far though. > > > > Thanks to everyone who gave early feedback on the issue. Holden has > offered > > to shepherd the SPIP. Comments are welcome here or on the doc. > > > > Thanks, > > Tornike > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
