Hello Antoine, I understand what Arrow is. Perhaps, you are confused by the phrases "Improved Arrow Flight" or "Arrow Flight++" that I used? These are metaphors intended to convey the essence of the table transfer protocols as succinctly as possible (though imperfectly), not the actual proposals to make table transfer protocols the next versions of Arrow Flight.
I started this discussion in the Arrow dev list because there is probably a reasonably high concentration of interested audience here. FWIW, I thought that DataFusion would be a more fitting home for the development of these protocols (at least, initially) among any of the existing Apache projects. I chose this dev list rather than DataFusion's dev list because there is already Arrow Flight as a transfer protocol reference for discussion (rather than "DataFusion Flight"), and because all potentially interested people in the DataFusion dev list likely read this dev list anyway. Re: sending Parquet files over HTTP, it would not help to efficiently query/process data stored in ClickHouse's (Databend, Doris, Druid, DeepLake, Firebolt, Lance, Oxla, Pinot, QuestDB, SingleStore, etc.) native at-rest partition file formats. On Fri, 13 Sept 2024 at 16:43, Antoine Pitrou <anto...@python.org> wrote: > > Hello, > > I'm perplexed by this discussion. If you want to send highly-compressed > files over the network that is already possible: just send Parquet > files via HTTP(S) (or another protocol of choice). > > Arrow Flight is simply a *streaming* protocol that allows > sending/requesting the Arrow format over gRPC. The Arrow format was > never meant to be high-compression as it's primarily an in-memory > format, and only secondarily a transfer/storage format. > > I don't think it's reasonable to enlarge the scope of Arrow to > something else. > > Regards > > Antoine. > > > > On Tue, 10 Sep 2024 01:29:05 +0800 > Roman Leventov <leven...@apache.org> wrote: > > Hello Arrow community, > > > > I've recently proposed a family of "table transfer protocols": Table > Read, > > Table Write, and Table Replication protocols (provisional names) that I > > think can become a full-standing alternative to Iceberg for interop > between > > diverse databases/data warehouses and processing and ML engines: > > > https://engineeringideas.substack.com/p/table-transfer-protocols-improved. > > > > It appears to me that the "convergence on Iceberg" doesn't really serve > the > > users because the competition in the storage layer is inhibited and > > everyone is using the "lowest common denominator" format that has > > limitations. See more details about this argument at the link above. > > > > But why then everyone is so excited about Iceberg then and push databases > > to add Iceberg catalogs (essentially turning themselves into processing > > engines), in addition to their native storage formats? > > 1) Cheap storage -- not due to Iceberg per se, but due to object > storage. > > Snowflake, Redshift, Firebolt, Databend, and other DBs have their own > > object storage-first storage formats, too, and many other DBs can be > > configured with aggressive tiering to object storage. > > 2) Interop with different processing engines -- Arrow Flight provides > > this, too > > 3) Ability to lift and move data to another DB or cloud -- this is fair, > > but the mere ability of the DB to write Iceberg permits the data team to > > convert and move *if* they decide to do it, but before that, why wouldn't > > they use DB's native format, more optimised for this DB, the ingestion > > pattern, etc.? > > 4) *Parallel writing* -- processing engine's nodes can write parquet > files > > completely in parallel within an Iceberg transaction, Arrow Flight > doesn't > > permit this > > 5) *Reading compressed data* from object storage to external processing > > engines -- Arrow Flight requires convert everything into Arrow before > > sending. If the processing job requires to pull a lot of Parquet > > files/columns (even after filtering partitions) converting them all to > > Arrow before sending as the minimum can increase the job latency quite a > > bit, as the maximum will cost much more money for egress. > > > > I cannot think of any other substantial reason (when compared with DBs > > rather than with Hive tables). If this is the case, Arrow Flight is not > yet > > a full-standing alternative to Iceberg catalogs for these three reasons: > > 1) *No parallel writes* > > 2) *Inability to send the table column data in the on-disk compressed > > formats* (rather than Arrow), and > > 3) A few other provisions in the protocol design that don't make it easy > > to control/balance the load of DB nodes, failover to replica nodes, the > > ability to re-request partition's endpoint from the server "in the same > > query" (that is, at the same consistency snapshot), etc. > > > > These are the things that I tried to address in Table Read > > < > https://engineeringideas.substack.com/i/148153268/table-read-protocol-walkthrough > > > > and Table Write > > < > https://engineeringideas.substack.com/i/148485754/use-cases-for-table-write-protocol > > > > protocols. > > > > So, I think the vendors of DBs like ClickHouse, Databend, Druid, Doris, > > InfluxDB, StarRocks, SingleStore, and others might be interested in these > > new protocols to gain foothold because it would enable these DBs to > compete > > on storage formats, and indexing/ingestion approaches, rather than be > > reduced to "processing engines for Iceberg", where there are already far > > too many other options. > > > > Specialised and "challenger" processing engine vendors might be > interested, > > too, as table transfer protocols that I proposed are kind of "Arrow > > Flight++", maybe there is a room for something like "DataFusion > Ballista++" > > where table transfer protocols are used as source and sink protocols > > instead of Arrow Flight. > > > > I'm interested in developing these protocols, but it requires some > > preliminary interest from vendors to validate the idea. Other feedback is > > welcome, too. > > > > > >
