Hi Junbo, What's 0.3.0? Since fluss-rust will be merged into fluss repo, maybe fluss release-1.0?
Best, Hongshun On Fri, Jul 3, 2026 at 11:27 AM Junbo Wang <[email protected]> wrote: > Thanks Forward and Hongshun for the thoughtful input. > > On ForwardXu's question about the target release — I've been thinking a > bit about the 0.3.0 scope, and would like to share my personal take, mostly > as a starting point for discussion. > > A possible 0.3.0 scope > > FIP-40: consolidate fluss-rust into apache/fluss. thanks to Anton for > already opening the PR-3401 <https://github.com/apache/fluss/pull/3401> > DataFusion integration adapter (Theme 1), as a fluss-rust submodule. > Lake + log union read, Paimon-first (Theme 2), as a separate submodule so > lake-side dependencies stay isolated from the core. S > erver-side filter pushdown (Theme 3) — would be great to have if capacity > allows, but I'd suggest keeping it optional rather than a hard requirement > for 0.3.0. > > Gateway — perhaps outside 0.3.0 > > Once fluss-rust is consolidated into apache/fluss, it might make sense for > the gateway to live as its own module depending on fluss-rust, rather than > being tied to the 0.3.0 release train. That would also give the auth/authz > questions Hongshun raised some room to be designed in a dedicated FIP. > > Just my personal thinking — happy to adjust based on what others feel is > right. > > > Best regards, > Junbo Wang > > > 2026年7月2日 11:47，Forward Xu <[email protected]> 写道： > > > > Hi Anton, > > > > Thanks for putting this together, and thanks to everyone who landed the > > previous roadmap. Framing the next phase around "fluss-rust as a > > first-class analytical query surface" makes a lot of sense to me — it > > builds naturally on the analytical primitives we just shipped. A few > > thoughts, roughly following your structure. > > > > *Theme 1 — DataFusion integration adapter [MUST-HAVE]* +1. I strongly > agree > > it should be an *adapter over the Rust core*, not a bundled engine. > Keeping > > fluss-datafusion as a standalone crate exposing TableProvider + > > CatalogProvider/SchemaProvider keeps the dependency direction clean and > > lets any DataFusion-based consumer (or the gateway) opt in. Mapping > > projection/filter/limit pushdown onto the existing access paths (PK > > equality → lookup, bucket-key prefix → prefix lookup, LIMIT → bounded > scan, > > else log scan) is the right approach. One thing worth nailing down early > is > > the exactness contract we report back to DataFusion ( > > TableProviderFilterPushDown::Exact vs Inexact), since getting that wrong > > silently drops or double-applies filters. > > > > *Theme 2 — Lake + log union read (Paimon-first) [MUST-HAVE]* +1, and > > Paimon-first is the right call — reusing paimon-rust by wrapping an > > existing reader is low-risk, and the lake-snapshot-records-log-offset > > design gives us a clean stitch point. Starting bounded-batch and > deferring > > streaming also de-risks correctness of the log-offset boundary before we > > add continuous reads. The main correctness edge I'd like to see covered > is > > the boundary semantics for PK tables when applying newer log records over > > the lake snapshot (dedup/ordering at the tiered offset). > > > > *Theme 3 — Foundations / server-side filter pushdown [STRETCH]* Agree > this > > is a stretch, but it's the item with the highest leverage since it prunes > > both lake and log scans and unblocks the gateway's filter pushdown. > > Cross-checking against the Java PredicateConverter semantics is important > > so both clients behave identically. If capacity allows, I'd lean toward > > pulling at least the simple value-predicate cases forward, since Themes 1 > > and 4 both benefit. > > > > *Theme 4 — Gateway [STRETCH]* The "REST is the no-regret floor, SQL is > the > > ceiling deferred until Themes 1–2" framing is a good way to think about > it. > > Keeping it a thin, stateless HTTP frontend over the existing client > > primitives (one endpoint per primitive) sounds right and keeps scope > > contained. > > > > *Open questions* > > > > - *Themes 1–2 must-have, rest stretch — reasonable?* Yes, ordering > looks > > right to me. Nothing obviously missing. > > - *Paimon first, Iceberg later?* Agree, provided the union-read path is > > written against a lake abstraction so Iceberg/Lance slot in the same > way > > rather than requiring a rewrite. > > - *Pin one arrow/DataFusion version across core, union-read/lake, and > > DF?* Strong +1 — we should pin a single Arrow/DataFusion version > > workspace-wide; version skew between these crates is a common and > painful > > source of breakage. > > - *Separate crates: lake kernel, DataFusion adapter, gateway?* +1 to > > separate crates. It matches the "adapter, not bundled engine" > principle and > > keeps the DataFusion dependency out of the core for consumers that > don't > > need it. > > > > One meta-question: do we have a rough sense of the target release > (0.3.0?) > > these must-haves land in, so we can size the Theme 3/4 stretch work > > accordingly? > > > > Thanks again — happy to help on the DataFusion adapter side. > > > > Best, > > > > ForwardXu > > > > Junbo Wang <[email protected]> 于2026年7月1日周三 22:41写道： > > > >>> I'd also like us to focus on closing the gap with the Java client on > >> predicate pushdown. > >> Agreed — predicate pushdown is a great optimization, and the read path > >> will benefit significantly from it. > >> > >> > >> > >> One thing that was missing in upstream paimon-rust was an > >> object-store-safe existence check in the filesystem catalog. It checked > >> whether database/table directories “exist”, which works on local > >> filesystems but can fail on S3/OSS because those directories are just > >> prefixes, not real objects. I forked it to fix that, so lake reads can > >> reliably open Paimon tables from object storage. > >> > >> Thanks again, Anton, for putting this roadmap together! I think we could > >> share it at the July 3rd community meeting. I've also been working on > some > >> designs around lake kernel read and the Fluss Gateway REST API — happy > to > >> discuss those as well if time allows. > >> > >> > >> Best regards, > >> Junbo Wang > >> > >>> 2026年6月25日 07:23，Anton Borisov <[email protected]> 写道： > >>> > >>> Hi Yuxia and Junbo, > >>> > >>> Yuxia, on PK changelog reads across the bindings: I have already > >>> prepared record-mode CDC for Rust, Python and C++. It's small, and > >>> the PK union read in Theme 2 needs it anyway. > >>> > >>> Junbo, I looked through your prototype branch. It is clear that a lot > >>> of Themes 1, 2 and 4 already exist there as working code and it does > >>> seem to answer two of the open questions: the three-way crate split > >>> (lake kernel / adapter / gateway), and pinning a single > Arrow/DataFusion > >>> version at the workspace level. +1 to both. > >>> > >>> Two separate FIPs with detailed design for REST/SQL gateway parts - > >> makes sense. > >>> > >>> One thing I noticed and want to discuss: > >>> The "lake read" currently depends on a fork of paimon-rust, can you > share > >>> what was missing in paimon-rust? > >>> > >>> I'd also like us to focus on closing the gap with the Java client on > >>> predicate pushdown. > >>> In the interim we can push predicates into the lake (Paimon already > >>> supports it) and apply them as a filter pass over the real-time log > >>> tail. > >>> > >>> Good to hear you want to drive this forward, given the prototype, that > >>> makes sense to me. I am happy to review the design and following PRs. > >>> > >>> Looking forward to the write-up with your findings. > >>> > >>> -- Anton > >>> > >>> вт, 23 июн. 2026 г. в 15:10, Junbo Wang <[email protected]>: > >>>> > >>>> Thanks Anton for kicking this off — funnily enough, I'd been arriving > >> at very similar conclusions while prototyping on my side, so let me > share a > >> bit of what came out of those experiments. > >>>> > >>>>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] > >>>> > >>>> Agree. Having the DataFusion integration live in fluss-rust would make > >> it reusable across more downstream projects. One thing we might want to > >> keep in mind: it would probably be worth aligning the DataFusion and > Arrow > >> versions with pg-datafusion so the whole stack stays on a consistent > set of > >> versions. > >>>> > >>>>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] > >>>> > >>>> Agree. I'd lean toward putting this in a separate fluss-lake module > >> that exposes a client capable of doing the union read over the lake and > the > >> Fluss log. That way the lake-side dependencies stay isolated in their > own > >> module and don't bleed into the core fluss-rust code. One thing that > might > >> be worth flagging here: the lake side already supports predicate > pushdown, > >> while Fluss doesn't yet — so we'd probably think about how the union > read > >> handles that asymmetry in the interim. > >>>> > >>>> THEME 4: GATEWAY [STRETCH] — A thin HTTP frontend over the client > >>>> > >>>> I wonder if we could expand on this one a bit. My suggestion would be > >> to split the SQL REST surface and the plain REST API into two separate > >> FIPs. A read/write REST API on its own — somewhat in the spirit of > >> kafka-rest, covering reads, writes, and metadata management — would > already > >> be a quick win for broadening Fluss's data ingestion/access reach, and > it > >> doesn't depend on anything else. The SQL REST surface, on the other > hand, > >> has to wait on Themes 1–2, so it might make sense to split them and ship > >> the plain REST API first. > >>>> > >>>> > >>>> > >>>> Also, it might make sense to have the gateway as a separate crate > >> inside the Fluss project itself — especially since fluss-rust will be > >> moving into the Fluss repo anyway. > >>>> > >>>> On a personal note, this is something I'm really excited about and > >> would love to help push forward. I'll share a write-up of what came out > of > >> my demo experiments with the community over the next few weeks. > >>>> > >>>> > >>>> Best regards, > >>>> Junbo Wang > >>>> > >>>>> 2026年6月23日 20:55，Yuxia Luo <[email protected]> 写道： > >>>>> > >>>>> Thanks for putting this together. +1 on the overall framing. My > >> answers to the open questions, plus one addition: > >>>>> > >>>>> Open questions: > >>>>> - Themes 1-2 as must-have, rest stretch: agree, the ordering makes > >> sense to me. > >>>>> - Paimon first, Iceberg later: +1. Reusing paimon-rust is the > >> pragmatic path, and the union-read design generalizes to Iceberg the > same > >> way later. > >>>>> - Pin one arrow/DataFusion version across core, union-read/lake, and > >> DF: yes, we should pin a single version. Letting these drift will cause > >> painful Arrow type/ABI mismatches across the crate boundary, so a shared > >> pinned version (bumped deliberately, not per-crate) is worth the > discipline. > >>>>> - Separate crates (lake kernel, DataFusion adapter, gateway): +1 on > >> separate crates. It keeps the dependency surface clean - the adapter and > >> gateway are optional consumers, and the lake kernel shouldn't drag > >> DataFusion into anyone who only needs the core read path. > >>>>> > >>>>> One addition I'd like to propose for the scope: > >>>>> PK-table CHANGELOG read, and expose it across the bindings - > >> specifically C++ and Python - so non-Rust CDC consumers can subscribe. > >>>>> > >>>>> On 2026/06/22 14:49:52 Anton Borisov wrote: > >>>>>> Hi all, > >>>>>> > >>>>>> With the previous roadmap wrapping up - complex types > >>>>>> (Array/Row/Map/nesting), limit and prefix scan, schema-aware KV > >>>>>> decoding, the metrics framework, and write optimisations, thanks to > >>>>>> everyone who contributed and reviewed, I'd like to open discussion > on > >>>>>> the roadmap. > >>>>>> > >>>>>> I propose framing around a single goal: make fluss-rust a > first-class > >>>>>> analytical query surface - a Rust-native SQL/DataFrame path over > Fluss > >>>>>> (DataFusion, and through it Polars/DuckDB/gateways), reading the > lake > >>>>>> tier at scale, built on the analytical primitives we just landed. > >>>>>> > >>>>>> As before, I grouped the items into themes with an initial > must-have / > >>>>>> stretch positioning. Please push back where you disagree. > >>>>>> > >>>>>> THEME 1: DATAFUSION INTEGRATION ADAPTER [MUST-HAVE] > >>>>>> A standalone fluss-datafusion crate exposing TableProvider + Catalog > >>>>>> over the existing client, so SELECT ... FROM . works from any > >>>>>> DataFusion-based engine. It should be framed as an integration > adapter > >>>>>> over the Rust core, not a bundled engine. A gateway (FIP-32) or any > >>>>>> analytical consumer can use it or not. > >>>>>> > >>>>>> Scope: TableProvider + CatalogProvider/SchemaProvider backed by the > >>>>>> metadata path. > >>>>>> > >>>>>> Map DataFusion's projection/filter/limit pushdown onto the client's > >>>>>> existing access paths: a full primary-key equality becomes a > lookup, a > >>>>>> bucket-key prefix becomes a prefix lookup, LIMIT becomes a bounded > >>>>>> scan, otherwise a log scan. Filters an access path fully satisfies > are > >>>>>> reported exact, the rest are left for DataFusion to apply > (server-side > >>>>>> filter pushdown is Theme 3). > >>>>>> > >>>>>> THEME 2: LAKE + LOG UNION READ (PAIMON-FIRST) [MUST-HAVE] > >>>>>> A client-side read that presents a lake-enabled table as one table. > >>>>>> Its history is tiered to the lake (Paimon) and recent writes are > still > >>>>>> in the Fluss log, the lake snapshot records the log offset it > covers, > >>>>>> so we read the lake up to that offset and the Fluss log past it, > then > >>>>>> combine. > >>>>>> Scope: > >>>>>> - Per bucket: the lake snapshot plus the Fluss log past the tiered > >> offset. > >>>>>> - Log tables: concatenate the two. PK tables: apply the newer log > >>>>>> records to get current state. > >>>>>> - Paimon first, reusing paimon-rust, Iceberg/Lance later the same > >>>>>> way. The DataFusion adapter reads lake-enabled tables through this > >>>>>> path. > >>>>>> - Bounded batch first, streaming later > >>>>>> > >>>>>> Reusing paimon-rust means wrapping an existing reader, so it's > rather > >>>>>> feasible and straightforward. > >>>>>> > >>>>>> THEME 3: FOUNDATIONS [STRETCH] > >>>>>> Server-side value-predicate (filter) pushdown. The proto field > exists, > >>>>>> but the client currently sends none. This prunes lake/log scans and > >>>>>> unblocks the gateway's filter pushdown (the open question I raised > for > >>>>>> 0.2.0), cross-checked against the Java PredicateConverter semantics. > >>>>>> > >>>>>> THEME 4: GATEWAY [STRETCH] > >>>>>> A thin HTTP frontend over the client we already have independent of > >>>>>> Themes 1–3. It's how non-Rust/non-SQL clients (TS/JS, microservices, > >>>>>> curl) write to Fluss and do simple key access. > >>>>>> - Write: upsert/append, delete, create/drop tables, metadata. The > >>>>>> mature client write path essentially. > >>>>>> - Read: point lookup, prefix lookup, bounded log / CDC poll - one > >>>>>> endpoint per existing primitive, bounded and stateless. > >>>>>> The tradeoff: no joins, filters, or aggregations - that's the SQL > >>>>>> surface, and it needs the adapter and probably will be an ongoing > >>>>>> effort to optimise later. > >>>>>> - If Themes 1–2 ship, we unlock SQL frontends on top - PostgreSQL > >>>>>> via datafusion-postgres, as one example > >>>>>> In short: REST is the no-regret floor, SQL is the ceiling deferred > >>>>>> until Themes 1–2. > >>>>>> > >>>>>> PARALLEL TRACK: ELIXIR + BINDING EXPOSURE > >>>>>> Continue Elixir parity and expose newly-landed primitives (prefix > >>>>>> lookup, etc.) across Python / C++ / Elixir. > >>>>>> > >>>>>> GOVERNANCE TRACK > >>>>>> FIP-40: Consolidate apache/fluss-rust into apache/fluss - PR is > ready, > >>>>>> looks pretty good, waiting for a good moment to rebase/merge. > >>>>>> > >>>>>> OPEN QUESTIONS > >>>>>> - Themes 1–2 must-have, the rest stretch. Reasonable? Anything > >>>>>> missing or mis-ordered? > >>>>>> - Paimon first, Iceberg later? > >>>>>> - Pin one arrow/DataFusion version across core, union-read/lake, and > >> DF? > >>>>>> - Separate crates: lake kernel, DataFusion adapter, gateway? > >>>>>> > >>>>>> Looking forward to feedback. > >>>>>> > >>>>>> -- Anton > >>>>>> > >>>> > >> > >> > >
