Hi all, I'd like to start a discussion on a SPIP to add a built-in ADBC (Arrow Database Connectivity) data source to Spark, alongside the existing JDBC one.
- SPIP doc: https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY - JIRA: https://issues.apache.org/jira/browse/SPARK-60255 - Earlier discussion: https://github.com/apache/spark/issues/54603 - Prototype: https://github.com/tokoko/spark-adbc Motivation JDBC returns data one row at a time, and Spark reads each value with a separate getXxx call. Many analytical databases store and send data in columns. Reading them through JDBC means converting columns to rows in the driver, reading each value one by one in Spark, and then converting back to columns again if Comet or Gluten is used. ADBC drivers return Arrow record batches instead. Spark can wrap those as ColumnarBatches through ArrowColumnVector, almost without copying. On the prototype, reads from columnar databases are 3-10x faster than JDBC, and the gap grows on wider tables. What's proposed - A new optional "adbc" format. There are no public API changes, and the JDBC source is unchanged. ADBC complements JDBC; it doesn't replace it. - Pushdown of column pruning, filters, limit, Top-N and aggregates (COUNT/SUM/MIN/MAX/AVG). - Two kinds of partitioning that can be combined: - JDBC-style range partitioning (partitionColumn/lowerBound/upperBound/numPartitions). - Driver partitioning through ADBC's executePartitioned, where each descriptor the driver returns becomes one Spark task. Flight SQL servers already split results this way. - Appending writes through ADBC bulk ingest, one transaction per task where the driver supports it. This is the same at-least-once guarantee as the JDBC writer. - No per-database dialect classes in Spark. The driver reports the few syntax choices pushdown needs through GetInfo/SqlInfo, and filter values are sent as bound Arrow parameters, not literals. As a second path, Substrait plans can be handed to drivers that translate them to SQL themselves. Out of scope for the first version: atomic job-level writes and Structured Streaming. Upstream dependencies. Two Arrow PRs would improve this. Neither blocks a first version: - apache/arrow#49796: SqlInfo codes for limit/offset syntax, NULLS ordering, LIKE escaping and parameter markers. - apache/arrow-adbc#4317: a partitioned bulk ingest API. It maps directly onto Spark's write commit protocol, which would make writes atomic across the whole job. The prototype currently runs a shared test suite against 13 engines: PostgreSQL, MySQL, MSSQL, DuckDB, ClickHouse, Trino, Presto, DataFusion, SingleStore, Exasol, Oracle, Flight SQL (GizmoSQL) and Spark Connect. Longer-term direction The first goal is narrow: a faster alternative to JDBC for databases that ship ADBC drivers. Longer term, I think ADBC could also become a general way for non-JVM sources to integrate with Spark. ADBC already covers catalog metadata (GetObjects, GetTableSchema), driver-planned partitioned reads, and bulk ingest, with partitioned ingest proposed in apache/arrow-adbc#4317. Substrait pushdown would let a driver accept more than SQL. Together, these could let catalogs and table formats with an ADBC driver plug into Spark, for example through a TableCatalog backed by GetObjects, without a Spark-specific connector. To try the idea out, I've put together a small proof of concept: an ADBC driver for Iceberg built on iceberg-rust (https://github.com/tokoko/adbc-iceberg). It's early and experimental so far though. Thanks to everyone who gave early feedback on the issue. Holden has offered to shepherd the SPIP. Comments are welcome here or on the doc. Thanks, Tornike
