I think discussion on the doc has settled if you want to move it forward to a VOTE Tornike?
On 2026-10-25 13:28 UTC Tornike Gurgenidze wrote: > Hi all, > > I'd like to start a discussion on a SPIP to add a built-in ADBC (Arrow > Database Connectivity) data source to Spark, alongside the existing JDBC > one. > > - SPIP doc: > https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY > - JIRA: https://issues.apache.org/jira/browse/SPARK-60255 > - Earlier discussion: https://github.com/apache/spark/issues/54603 > - Prototype: https://github.com/tokoko/spark-adbc > > Motivation > > JDBC returns data one row at a time, and Spark reads each value with a > separate getXxx call. Many analytical databases store and send data in > columns. Reading them through JDBC means converting columns to rows in the > driver, reading each value one by one in Spark, and then converting back to > columns again if Comet or Gluten is used. ADBC drivers return Arrow record > batches instead. Spark can wrap those as ColumnarBatches through > ArrowColumnVector, almost without copying. On the prototype, reads from > columnar databases are 3-10x faster than JDBC, and the gap grows on wider > tables. > > What's proposed > > - A new optional "adbc" format. There are no public API changes, and the > JDBC source is unchanged. ADBC complements JDBC; it doesn't replace it. > - Pushdown of column pruning, filters, limit, Top-N and aggregates > (COUNT/SUM/MIN/MAX/AVG). > - Two kinds of partitioning that can be combined: > - JDBC-style range partitioning > (partitionColumn/lowerBound/upperBound/numPartitions). > - Driver partitioning through ADBC's executePartitioned, where each > descriptor the driver returns becomes one Spark task. Flight SQL servers > already split results this way. > - Appending writes through ADBC bulk ingest, one transaction per task where > the driver supports it. This is the same at-least-once guarantee as the > JDBC writer. > - No per-database dialect classes in Spark. The driver reports the few > syntax choices pushdown needs through GetInfo/SqlInfo, and filter values > are sent as bound Arrow parameters, not literals. As a second path, > Substrait plans can be handed to drivers that translate them to SQL > themselves. > > Out of scope for the first version: atomic job-level writes and Structured > Streaming. > > Upstream dependencies. Two Arrow PRs would improve this. Neither blocks a > first version: > > - apache/arrow#49796: SqlInfo codes for limit/offset syntax, NULLS > ordering, LIKE escaping and parameter markers. > - apache/arrow-adbc#4317: a partitioned bulk ingest API. It maps directly > onto Spark's write commit protocol, which would make writes atomic across > the whole job. > > The prototype currently runs a shared test suite against 13 engines: > PostgreSQL, MySQL, MSSQL, DuckDB, ClickHouse, Trino, Presto, DataFusion, > SingleStore, Exasol, Oracle, Flight SQL (GizmoSQL) and Spark Connect. > > Longer-term direction > > The first goal is narrow: a faster alternative to JDBC for databases that > ship ADBC drivers. Longer term, I think ADBC could also become a general > way for non-JVM sources to integrate with Spark. ADBC already covers > catalog metadata (GetObjects, GetTableSchema), driver-planned partitioned > reads, and bulk ingest, with partitioned ingest proposed in > apache/arrow-adbc#4317. Substrait pushdown would let a driver accept more > than SQL. Together, these could let catalogs and table formats with an ADBC > driver plug into Spark, for example through a TableCatalog backed by > GetObjects, without a Spark-specific connector. To try the idea out, I've > put together a small proof of concept: an ADBC driver for Iceberg built on > iceberg-rust (https://github.com/tokoko/adbc-iceberg). It's early and > experimental so far though. > > Thanks to everyone who gave early feedback on the issue. Holden has offered > to shepherd the SPIP. Comments are welcome here or on the doc. > > Thanks, > Tornike > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
