Thanks Anton for the deep review — great catches, all points are now updated in the FIP-49. 1. the new cooperative backpressure / request timeout on writes Both need client-side capabilities that fluss-rust doesn't have yet. I've filed issues to implement them — #3855 <https://github.com/apache/fluss/issues/3855>(cooperative backpressure, client side of #3463) and #3861 <https://github.com/apache/fluss/issues/3861> (a delivery.timeout.ms-style delivery bound) — under the umbrella #3854 <https://github.com/apache/fluss/issues/3854>, and the REST gateway will build on these capabilities.On top of them, the FIP-49 will run every write under an end-to-end deadline (gateway.rest.write.request-timeout, default 30s), with client delivery bounded below it. Pressure shows up as bounded latency, observable could via re-exported client metrics; hard rejections surface as a retriable entry-level storage_backpressure, so callers retry just those entries. At the deadline, in-flight entries are reported timeout (indeterminate); never-sent entries fail deterministically.
2. connections in service mode / per-user authorization The FIP-49 design is simply two identity modes. service: one shared connection per cluster — so you're right, connection.max can't fire there; it and idle-timeout are scoped to user mode only. user: multiple connections per cluster, one per user id, using impersonation — the connection authenticates as a super-user and carries the end user as the SASL/PLAIN authorization id, so Fluss authorizes each request as that user against its own ACLs. That is the per-user authorization path; prerequisite is #3857 <https://github.com/apache/fluss/issues/3837>, and user mode fails at startup until it's integrated. 4.which tables get per-row results? That was a wording error in the FIP — the REST surface is JSON-only and all writes are row-by-row, so every table gets per-row results. Corrected: the append_arrow_batch path is dropped from this FIP (deferred to a future columnar content type); the accumulator still batches per bucket, so nothing is lost. 5.duplicates on client retry The FIP-49 design is at-least-once — HTTP retries are not deduplicated. KV tables converge per key; for log tables dedup belongs downstream (and retry only the entries in failures). Same stance as Kafka REST Proxy. 6. type names Good catch — that was an oversight. Fixed: the FIP now uses the DataTypeJsonSerde vocabulary verbatim (INTEGER,TIMESTAMP_WITH_LOCAL_TIME_ZONE, ...), so REST schema objects are identical to the persisted serialization and no mapping exists anywhere. All changes are updated on the Confluence page.Thanks again Anton for the thorough review — these suggestions were really helpful. Best regards, Junbo Wang > On 4 Aug 2026, at 14:04, Junbo Wang <[email protected]> wrote: > > Thanks for the MKO's advice — good point! We started with Prometheus pull for > the MVP mainly for simplicity — it works out of the box and aligns with how > Fluss servers expose metrics today. That said, it doesn't preclude OTel: a > Collector can scrape the /metrics endpoint and re-export via OTLP, and we > plan to keep the config extensible so a native OTLP exporter can be added > later. > > > Thanks for the great questions, Liting Liu. > > > Why Rust instead of Java/Netty? > Three reasons: (a) the gateway is a proxy-style component where tail latency > and footprint matter — Rust gives GC-free predictable latency and a much > smaller memory footprint as a single static binary; (b) ecosystem fit — Fluss > data is Arrow-native, and the Rust stack (arrow-rs, DataFusion, pgwire) keeps > a zero-copy path end to end; (c) the gateway also helps drive the fluss-rust > client to production readiness and grow the Rust ecosystem. > > > a query failure or OOM could potentially affect the entire Gateway. > > Query execution will run on a dedicated tokio runtime with a bounded thread > pool, so it can't starve the write path. For memory, DataFusion's built-in > MemoryPool sets per-query budgets — an over-limit query fails alone, not the > process. We'll also add concurrency limits and query timeouts. If heavy > workloads ever emerge, a separate query worker remains an option. > > > > > a lightweight lookup REST API backed directly by the Fluss client, > > Agreed — the initial read path is a lightweight lookup API backed directly by > the Fluss client, no DataFusion. DataFusion comes later during FIP-32 > implementation, and only for lightweight queries (filter/projection/limit). > > > -- > Best regards, > Junbo Wang > > > Anton Borisov <[email protected] <mailto:[email protected]>> > 于2026年8月4日周二 10:30写道： >> Hi Junbo, >> >> Nice proposal, thank you for driving it! >> I like the direction overall, +1 >> >> My questions below: >> >> 1. Have you seen the new cooperative backpressure? >> #3463 [1] went in on 31 July, a day after this thread started. Tablet >> servers now >> send a per-bucket pressure value on PutKv responses, the client throttles on >> that by up to 3 seconds by default, and the server can reject with a >> retriable >> StorageBackpressureException when RocksDB is close to stalling. >> >> Inside an HTTP request that becomes invisible latency, the caller waits and >> isn't told why. It also looks like the thing your rate limiting wants to be >> driven by. A 429 with Retry-After says "slow down" in a way HTTP clients >> already understand, and a fixed requests/sec limit can't track what the >> cluster is actually doing. Worth pulling into the FIP? >> >> 2. What is a "connection" in service mode, and how many are there? >> The page says service mode uses "one shared pool per cluster", and >> connection.max is a "total connection cap (exceeded -> 429)". But it doesn't >> say what a connection is, or how many service mode opens. >> >> It matters because a Fluss Connection isn't a socket. It's documented as >> heavy-weight to create, thread-safe and meant to be shared, and each one >> carries a 64 MB write buffer, a sender task, a writer id and sockets to every >> tablet server. If service mode really is one shared connection, then the 512 >> cap and its 429 never fire, and it'd be clearer to say so. If it's many, >> they'll batch worse than one would, since each accumulator sees only a >> fraction of the traffic. Kafka REST Proxy shares a single producer across >> requests for that reason, and only goes per-principal when identities differ. >> >> 3. Is there a request timeout on writes? >> The client retries forever by default, and "leader not available" is one of >> the errors it retries. So the row in your example wouldn't fail, it would >> wait. And if the gateway stops waiting and reports a failure, does the row >> stay queued and land later? That would make "failed" sometimes mean >> "succeeded, just later". >> >> 4. Which tables get per-row results? >> For Log tables without a bucket key you use append_arrow_batch, which returns >> one result for the whole batch and stops at the first failure. I don't think >> per-row reporting is possible on that path. Is it meant to be all-or-nothing >> there? >> >> 5. What stops a duplicate when a client retries? >> Our idempotence catches the writer resending its own batch. A retried HTTP >> request is a different thing - the rows arrive fresh, so nothing links them >> to >> the first attempt. The writer id also belongs to one process, so a second >> gateway can't pick up where the first left off. >> >> 6. Is per-user authorization on the roadmap? >> The page says Fluss-side ACLs see only the gateway account, so every request >> reaches the cluster as gateway_svc and the server can't tell one caller from >> another. Is something planned for later: which principals may write to which >> tables, or is the gateway meant only for deployments where one shared >> identity is fine? >> >> Last one, the type names. Fluss spells types two ways: INT and TIMESTAMP_LTZ >> in SQL, INTEGER and TIMESTAMP_WITH_LOCAL_TIME_ZONE in DataTypeJsonSerde. The >> FIP uses the SQL names with the serde's JSON shape, so I don't think the >> examples parse as written. >> >> Which did you intend? If it's the SQL names, the gateway has to convert them >> before calling CreateTable, and that mapping is worth writing into the FIP. >> >> Thank you! >> >> -- Anton >> >> [1] https://github.com/apache/fluss/pull/3463 >> >> пн, 3 авг. 2026 г. в 16:02, David Reger <[email protected] >> <mailto:[email protected]>>: >> > >> > I think this also relates to one of the architectural discussions we've >> > recently had around FIP-32 [1]. >> > >> > My understanding is that there is a useful distinction between *native >> > Fluss operations* and *SQL/query execution*. >> > >> > For example, writes, metadata operations, and potentially lightweight >> > native reads (such as point lookups) could continue to build directly on >> > the fluss-rust foundation, whereas DataFusion would primarily serve the >> > SQL/query path. This would allow the REST gateway to remain lightweight >> > while keeping the query layer reusable for SQL-based protocols in the >> > future. >> > >> > I don't think this necessarily changes the scope of FIP-49, but I do think >> > it's an important architectural distinction as we evolve the gateway beyond >> > the initial REST write MVP. >> > >> > [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x >> > >> > Best regards, >> > >> > David >> > >> > Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev < >> > [email protected] <mailto:[email protected]>>: >> > >> > > Hi， Junbo >> > > >> > > >> > > Thanks for the detailed proposal. I have a couple of questions about the >> > > technology choice and the future SQL path. >> > > >> > > For the current scope—REST writes and metadata/DDL—have we considered >> > > implementing the Gateway with a Java/Netty REST stack backed by the >> > > existing Fluss Java client? This would be similar in spirit to how Flink >> > > SQL Gateway reuses Flink’s native Java stack and Netty-based REST >> > > infrastructure. The Fluss Java client already supports append, upsert, >> > > delete, partial update, metadata operations, and lookup. It would be >> > > helpful if the FIP could compare this alternative with the proposed >> > > >> > > I also have a question about the future DataFusion integration. Is >> > > DataFusion expected to execute SQL queries inside the same Gateway >> > > process? >> > > If so, complex or resource-intensive queries could compete with REST >> > > ingestion and metadata operations for CPU and memory, and a query failure >> > > or OOM could potentially affect the entire Gateway. Do we plan to provide >> > > admission control, memory and concurrency limits, query >> > > timeout/cancellation, spilling, or process-level isolation? >> > > >> > > If the initial read use case is mainly primary-key or prefix lookup, >> > > could >> > > we expose a lightweight lookup REST API backed directly by the Fluss >> > > client, without introducing DataFusion? DataFusion seems more valuable >> > > when >> > > the intended scope includes general SQL capabilities such as scans, >> > > filters, aggregations, and joins? If that is the long-term goal, would it >> > > be preferable to keep complex SQL execution in a separate optional SQL >> > > service or worker process, so that it does not increase the failure >> > > domain >> > > of the REST write Gateway? >> > > >> > > >> > > From: Michael Koepf <[email protected] >> > > <mailto:[email protected]>> >> > > Date: Thursday, July 30, 2026 at 22:15 >> > > To: [email protected] <mailto:[email protected]> >> > > <[email protected] <mailto:[email protected]>> >> > > Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API >> > > >> > > Hi Juno, >> > > >> > > Thanks for the FIP, great initiative. >> > > >> > > A question that comes to my mind: Why not expose metrics via vendor >> > > neutral OTEL instead of limiting it to Prometheus? >> > > -- >> > > Best, >> > > MKO >> > > >> > > >> > > > On 30.07.2026, at 12:23, Forward Xu <[email protected] >> > > > <mailto:[email protected]>> wrote: >> > > > >> > > > Thanks junbo, >> > > > >> > > > The proposal looks good to me in general. +1 >> > > > >> > > > Best, >> > > > Foward >> > > > >> > > > Junbo Wang <[email protected] <mailto:[email protected]>> >> > > > 于2026年7月30日周四 11:49写道： >> > > > >> > > >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST >> > > API. >> > > >> >> > > >> The proposal is available here: >> > > >> >> > > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API >> > > >> >> > > >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] >> > > >> and >> > > >> does not conflict with it. Following the earlier suggestion to split >> > > >> the >> > > >> gateway work into focused FIPs, FIP-49 defines the REST write path and >> > > DDL >> > > >> / metadata management in detail, while REST read and the SQL path >> > > remain in >> > > >> the scope of FIP-32. >> > > >> >> > > >> In short, it proposes a standalone, stateless fluss-gateway process >> > > (Rust, >> > > >> axum + tokio, built on fluss-rust) providing: >> > > >> >> > > >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with >> > > >> per-row >> > > >> success/failure reporting >> > > >> Full database / table / partition DDL >> > > >> Pluggable client authentication, and a service / user (act-as) >> > > >> identity >> > > >> mode towards Fluss >> > > >> Multi-cluster routing, Prometheus metrics, a unified error model >> > > >> >> > > >> No Fluss server-side changes are required— the gateway accesses >> > > >> existing >> > > >> clusters purely as a fluss-rust client. >> > > >> >> > > >> Looking forward to your feedback and suggestions. >> > > >> >> > > >> >> > > >> >> > > >> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x >> > > >> >> > > >> >> > > >> Best regards, >> > > >> Junbo Wang >> > > >> >> > > >> >> > > > > > > >
