Hello,
I'm perplexed by this discussion. If you want to send highly-compressed files over the network that is already possible: just send Parquet files via HTTP(S) (or another protocol of choice). Arrow Flight is simply a *streaming* protocol that allows sending/requesting the Arrow format over gRPC. The Arrow format was never meant to be high-compression as it's primarily an in-memory format, and only secondarily a transfer/storage format. I don't think it's reasonable to enlarge the scope of Arrow to something else. Regards Antoine. On Tue, 10 Sep 2024 01:29:05 +0800 Roman Leventov <leven...@apache.org> wrote: > Hello Arrow community, > > I've recently proposed a family of "table transfer protocols": Table Read, > Table Write, and Table Replication protocols (provisional names) that I > think can become a full-standing alternative to Iceberg for interop between > diverse databases/data warehouses and processing and ML engines: > https://engineeringideas.substack.com/p/table-transfer-protocols-improved. > > It appears to me that the "convergence on Iceberg" doesn't really serve the > users because the competition in the storage layer is inhibited and > everyone is using the "lowest common denominator" format that has > limitations. See more details about this argument at the link above. > > But why then everyone is so excited about Iceberg then and push databases > to add Iceberg catalogs (essentially turning themselves into processing > engines), in addition to their native storage formats? > 1) Cheap storage -- not due to Iceberg per se, but due to object storage. > Snowflake, Redshift, Firebolt, Databend, and other DBs have their own > object storage-first storage formats, too, and many other DBs can be > configured with aggressive tiering to object storage. > 2) Interop with different processing engines -- Arrow Flight provides > this, too > 3) Ability to lift and move data to another DB or cloud -- this is fair, > but the mere ability of the DB to write Iceberg permits the data team to > convert and move *if* they decide to do it, but before that, why wouldn't > they use DB's native format, more optimised for this DB, the ingestion > pattern, etc.? > 4) *Parallel writing* -- processing engine's nodes can write parquet files > completely in parallel within an Iceberg transaction, Arrow Flight doesn't > permit this > 5) *Reading compressed data* from object storage to external processing > engines -- Arrow Flight requires convert everything into Arrow before > sending. If the processing job requires to pull a lot of Parquet > files/columns (even after filtering partitions) converting them all to > Arrow before sending as the minimum can increase the job latency quite a > bit, as the maximum will cost much more money for egress. > > I cannot think of any other substantial reason (when compared with DBs > rather than with Hive tables). If this is the case, Arrow Flight is not yet > a full-standing alternative to Iceberg catalogs for these three reasons: > 1) *No parallel writes* > 2) *Inability to send the table column data in the on-disk compressed > formats* (rather than Arrow), and > 3) A few other provisions in the protocol design that don't make it easy > to control/balance the load of DB nodes, failover to replica nodes, the > ability to re-request partition's endpoint from the server "in the same > query" (that is, at the same consistency snapshot), etc. > > These are the things that I tried to address in Table Read > <https://engineeringideas.substack.com/i/148153268/table-read-protocol-walkthrough> > and Table Write > <https://engineeringideas.substack.com/i/148485754/use-cases-for-table-write-protocol> > protocols. > > So, I think the vendors of DBs like ClickHouse, Databend, Druid, Doris, > InfluxDB, StarRocks, SingleStore, and others might be interested in these > new protocols to gain foothold because it would enable these DBs to compete > on storage formats, and indexing/ingestion approaches, rather than be > reduced to "processing engines for Iceberg", where there are already far > too many other options. > > Specialised and "challenger" processing engine vendors might be interested, > too, as table transfer protocols that I proposed are kind of "Arrow > Flight++", maybe there is a room for something like "DataFusion Ballista++" > where table transfer protocols are used as source and sink protocols > instead of Arrow Flight. > > I'm interested in developing these protocols, but it requires some > preliminary interest from vendors to validate the idea. Other feedback is > welcome, too. >
