> I'd also like us to focus on closing the gap with the Java client on > predicate pushdown. Agreed — predicate pushdown is a great optimization, and the read path will benefit significantly from it.
One thing that was missing in upstream paimon-rust was an object-store-safe existence check in the filesystem catalog. It checked whether database/table directories “exist”, which works on local filesystems but can fail on S3/OSS because those directories are just prefixes, not real objects. I forked it to fix that, so lake reads can reliably open Paimon tables from object storage. Thanks again, Anton, for putting this roadmap together! I think we could share it at the July 3rd community meeting. I've also been working on some designs around lake kernel read and the Fluss Gateway REST API — happy to discuss those as well if time allows. Best regards, Junbo Wang > 2026年6月25日 07:23，Anton Borisov <[email protected]> 写道： > > Hi Yuxia and Junbo, > > Yuxia, on PK changelog reads across the bindings: I have already > prepared record-mode CDC for Rust, Python and C++. It's small, and > the PK union read in Theme 2 needs it anyway. > > Junbo, I looked through your prototype branch. It is clear that a lot > of Themes 1, 2 and 4 already exist there as working code and it does > seem to answer two of the open questions: the three-way crate split > (lake kernel / adapter / gateway), and pinning a single Arrow/DataFusion > version at the workspace level. +1 to both. > > Two separate FIPs with detailed design for REST/SQL gateway parts - makes > sense. > > One thing I noticed and want to discuss: > The "lake read" currently depends on a fork of paimon-rust, can you share > what was missing in paimon-rust? > > I'd also like us to focus on closing the gap with the Java client on > predicate pushdown. > In the interim we can push predicates into the lake (Paimon already > supports it) and apply them as a filter pass over the real-time log > tail. > > Good to hear you want to drive this forward, given the prototype, that > makes sense to me. I am happy to review the design and following PRs. > > Looking forward to the write-up with your findings. > > -- Anton > > вт, 23 июн. 2026 г. в 15:10, Junbo Wang <[email protected]>: >> >> Thanks Anton for kicking this off — funnily enough, I'd been arriving at >> very similar conclusions while prototyping on my side, so let me share a bit >> of what came out of those experiments. >> >>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] >> >> Agree. Having the DataFusion integration live in fluss-rust would make it >> reusable across more downstream projects. One thing we might want to keep in >> mind: it would probably be worth aligning the DataFusion and Arrow versions >> with pg-datafusion so the whole stack stays on a consistent set of versions. >> >>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] >> >> Agree. I'd lean toward putting this in a separate fluss-lake module that >> exposes a client capable of doing the union read over the lake and the Fluss >> log. That way the lake-side dependencies stay isolated in their own module >> and don't bleed into the core fluss-rust code. One thing that might be worth >> flagging here: the lake side already supports predicate pushdown, while >> Fluss doesn't yet — so we'd probably think about how the union read handles >> that asymmetry in the interim. >> >> THEME 4: GATEWAY [STRETCH] — A thin HTTP frontend over the client >> >> I wonder if we could expand on this one a bit. My suggestion would be to >> split the SQL REST surface and the plain REST API into two separate FIPs. A >> read/write REST API on its own — somewhat in the spirit of kafka-rest, >> covering reads, writes, and metadata management — would already be a quick >> win for broadening Fluss's data ingestion/access reach, and it doesn't >> depend on anything else. The SQL REST surface, on the other hand, has to >> wait on Themes 1–2, so it might make sense to split them and ship the plain >> REST API first. >> >> >> >> Also, it might make sense to have the gateway as a separate crate inside the >> Fluss project itself — especially since fluss-rust will be moving into the >> Fluss repo anyway. >> >> On a personal note, this is something I'm really excited about and would >> love to help push forward. I'll share a write-up of what came out of my demo >> experiments with the community over the next few weeks. >> >> >> Best regards, >> Junbo Wang >> >>> 2026年6月23日 20:55，Yuxia Luo <[email protected]> 写道： >>> >>> Thanks for putting this together. +1 on the overall framing. My answers to >>> the open questions, plus one addition: >>> >>> Open questions: >>> - Themes 1-2 as must-have, rest stretch: agree, the ordering makes sense to >>> me. >>> - Paimon first, Iceberg later: +1. Reusing paimon-rust is the pragmatic >>> path, and the union-read design generalizes to Iceberg the same way later. >>> - Pin one arrow/DataFusion version across core, union-read/lake, and DF: >>> yes, we should pin a single version. Letting these drift will cause painful >>> Arrow type/ABI mismatches across the crate boundary, so a shared pinned >>> version (bumped deliberately, not per-crate) is worth the discipline. >>> - Separate crates (lake kernel, DataFusion adapter, gateway): +1 on >>> separate crates. It keeps the dependency surface clean - the adapter and >>> gateway are optional consumers, and the lake kernel shouldn't drag >>> DataFusion into anyone who only needs the core read path. >>> >>> One addition I'd like to propose for the scope: >>> PK-table CHANGELOG read, and expose it across the bindings - specifically >>> C++ and Python - so non-Rust CDC consumers can subscribe. >>> >>> On 2026/06/22 14:49:52 Anton Borisov wrote: >>>> Hi all, >>>> >>>> With the previous roadmap wrapping up - complex types >>>> (Array/Row/Map/nesting), limit and prefix scan, schema-aware KV >>>> decoding, the metrics framework, and write optimisations, thanks to >>>> everyone who contributed and reviewed, I'd like to open discussion on >>>> the roadmap. >>>> >>>> I propose framing around a single goal: make fluss-rust a first-class >>>> analytical query surface - a Rust-native SQL/DataFrame path over Fluss >>>> (DataFusion, and through it Polars/DuckDB/gateways), reading the lake >>>> tier at scale, built on the analytical primitives we just landed. >>>> >>>> As before, I grouped the items into themes with an initial must-have / >>>> stretch positioning. Please push back where you disagree. >>>> >>>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] >>>> A standalone fluss-datafusion crate exposing TableProvider + Catalog >>>> over the existing client, so SELECT ... FROM . works from any >>>> DataFusion-based engine. It should be framed as an integration adapter >>>> over the Rust core, not a bundled engine. A gateway (FIP-32) or any >>>> analytical consumer can use it or not. >>>> >>>> Scope: TableProvider + CatalogProvider/SchemaProvider backed by the >>>> metadata path. >>>> >>>> Map DataFusion's projection/filter/limit pushdown onto the client's >>>> existing access paths: a full primary-key equality becomes a lookup, a >>>> bucket-key prefix becomes a prefix lookup, LIMIT becomes a bounded >>>> scan, otherwise a log scan. Filters an access path fully satisfies are >>>> reported exact, the rest are left for DataFusion to apply (server-side >>>> filter pushdown is Theme 3). >>>> >>>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] >>>> A client-side read that presents a lake-enabled table as one table. >>>> Its history is tiered to the lake (Paimon) and recent writes are still >>>> in the Fluss log, the lake snapshot records the log offset it covers, >>>> so we read the lake up to that offset and the Fluss log past it, then >>>> combine. >>>> Scope: >>>> - Per bucket: the lake snapshot plus the Fluss log past the tiered offset. >>>> - Log tables: concatenate the two. PK tables: apply the newer log >>>> records to get current state. >>>> - Paimon first, reusing paimon-rust, Iceberg/Lance later the same >>>> way. The DataFusion adapter reads lake-enabled tables through this >>>> path. >>>> - Bounded batch first, streaming later >>>> >>>> Reusing paimon-rust means wrapping an existing reader, so it's rather >>>> feasible and straightforward. >>>> >>>> THEME 3: FOUNDATIONS [STRETCH] >>>> Server-side value-predicate (filter) pushdown. The proto field exists, >>>> but the client currently sends none. This prunes lake/log scans and >>>> unblocks the gateway's filter pushdown (the open question I raised for >>>> 0.2.0), cross-checked against the Java PredicateConverter semantics. >>>> >>>> THEME 4: GATEWAY [STRETCH] >>>> A thin HTTP frontend over the client we already have independent of >>>> Themes 1–3. It's how non-Rust/non-SQL clients (TS/JS, microservices, >>>> curl) write to Fluss and do simple key access. >>>> - Write: upsert/append, delete, create/drop tables, metadata. The >>>> mature client write path essentially. >>>> - Read: point lookup, prefix lookup, bounded log / CDC poll - one >>>> endpoint per existing primitive, bounded and stateless. >>>> The tradeoff: no joins, filters, or aggregations - that's the SQL >>>> surface, and it needs the adapter and probably will be an ongoing >>>> effort to optimise later. >>>> - If Themes 1–2 ship, we unlock SQL frontends on top - PostgreSQL >>>> via datafusion-postgres, as one example >>>> In short: REST is the no-regret floor, SQL is the ceiling deferred >>>> until Themes 1–2. >>>> >>>> PARALLEL TRACK: ELIXIR + BINDING EXPOSURE >>>> Continue Elixir parity and expose newly-landed primitives (prefix >>>> lookup, etc.) across Python / C++ / Elixir. >>>> >>>> GOVERNANCE TRACK >>>> FIP-40: Consolidate apache/fluss-rust into apache/fluss - PR is ready, >>>> looks pretty good, waiting for a good moment to rebase/merge. >>>> >>>> OPEN QUESTIONS >>>> - Themes 1–2 must-have, the rest stretch. Reasonable? Anything >>>> missing or mis-ordered? >>>> - Paimon first, Iceberg later? >>>> - Pin one arrow/DataFusion version across core, union-read/lake, and DF? >>>> - Separate crates: lake kernel, DataFusion adapter, gateway? >>>> >>>> Looking forward to feedback. >>>> >>>> -- Anton >>>> >>
