Hi Junbo, Nice proposal, thank you for driving it! I like the direction overall, +1
My questions below: 1. Have you seen the new cooperative backpressure? #3463 [1] went in on 31 July, a day after this thread started. Tablet servers now send a per-bucket pressure value on PutKv responses, the client throttles on that by up to 3 seconds by default, and the server can reject with a retriable StorageBackpressureException when RocksDB is close to stalling. Inside an HTTP request that becomes invisible latency, the caller waits and isn't told why. It also looks like the thing your rate limiting wants to be driven by. A 429 with Retry-After says "slow down" in a way HTTP clients already understand, and a fixed requests/sec limit can't track what the cluster is actually doing. Worth pulling into the FIP? 2. What is a "connection" in service mode, and how many are there? The page says service mode uses "one shared pool per cluster", and connection.max is a "total connection cap (exceeded -> 429)". But it doesn't say what a connection is, or how many service mode opens. It matters because a Fluss Connection isn't a socket. It's documented as heavy-weight to create, thread-safe and meant to be shared, and each one carries a 64 MB write buffer, a sender task, a writer id and sockets to every tablet server. If service mode really is one shared connection, then the 512 cap and its 429 never fire, and it'd be clearer to say so. If it's many, they'll batch worse than one would, since each accumulator sees only a fraction of the traffic. Kafka REST Proxy shares a single producer across requests for that reason, and only goes per-principal when identities differ. 3. Is there a request timeout on writes? The client retries forever by default, and "leader not available" is one of the errors it retries. So the row in your example wouldn't fail, it would wait. And if the gateway stops waiting and reports a failure, does the row stay queued and land later? That would make "failed" sometimes mean "succeeded, just later". 4. Which tables get per-row results? For Log tables without a bucket key you use append_arrow_batch, which returns one result for the whole batch and stops at the first failure. I don't think per-row reporting is possible on that path. Is it meant to be all-or-nothing there? 5. What stops a duplicate when a client retries? Our idempotence catches the writer resending its own batch. A retried HTTP request is a different thing - the rows arrive fresh, so nothing links them to the first attempt. The writer id also belongs to one process, so a second gateway can't pick up where the first left off. 6. Is per-user authorization on the roadmap? The page says Fluss-side ACLs see only the gateway account, so every request reaches the cluster as gateway_svc and the server can't tell one caller from another. Is something planned for later: which principals may write to which tables, or is the gateway meant only for deployments where one shared identity is fine? Last one, the type names. Fluss spells types two ways: INT and TIMESTAMP_LTZ in SQL, INTEGER and TIMESTAMP_WITH_LOCAL_TIME_ZONE in DataTypeJsonSerde. The FIP uses the SQL names with the serde's JSON shape, so I don't think the examples parse as written. Which did you intend? If it's the SQL names, the gateway has to convert them before calling CreateTable, and that mapping is worth writing into the FIP. Thank you! -- Anton [1] https://github.com/apache/fluss/pull/3463 пн, 3 авг. 2026 г. в 16:02, David Reger <[email protected]>: > > I think this also relates to one of the architectural discussions we've > recently had around FIP-32 [1]. > > My understanding is that there is a useful distinction between *native > Fluss operations* and *SQL/query execution*. > > For example, writes, metadata operations, and potentially lightweight > native reads (such as point lookups) could continue to build directly on > the fluss-rust foundation, whereas DataFusion would primarily serve the > SQL/query path. This would allow the REST gateway to remain lightweight > while keeping the query layer reusable for SQL-based protocols in the > future. > > I don't think this necessarily changes the scope of FIP-49, but I do think > it's an important architectural distinction as we evolve the gateway beyond > the initial REST write MVP. > > [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x > > Best regards, > > David > > Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev < > [email protected]>: > > > Hi， Junbo > > > > > > Thanks for the detailed proposal. I have a couple of questions about the > > technology choice and the future SQL path. > > > > For the current scope—REST writes and metadata/DDL—have we considered > > implementing the Gateway with a Java/Netty REST stack backed by the > > existing Fluss Java client? This would be similar in spirit to how Flink > > SQL Gateway reuses Flink’s native Java stack and Netty-based REST > > infrastructure. The Fluss Java client already supports append, upsert, > > delete, partial update, metadata operations, and lookup. It would be > > helpful if the FIP could compare this alternative with the proposed > > > > I also have a question about the future DataFusion integration. Is > > DataFusion expected to execute SQL queries inside the same Gateway process? > > If so, complex or resource-intensive queries could compete with REST > > ingestion and metadata operations for CPU and memory, and a query failure > > or OOM could potentially affect the entire Gateway. Do we plan to provide > > admission control, memory and concurrency limits, query > > timeout/cancellation, spilling, or process-level isolation? > > > > If the initial read use case is mainly primary-key or prefix lookup, could > > we expose a lightweight lookup REST API backed directly by the Fluss > > client, without introducing DataFusion? DataFusion seems more valuable when > > the intended scope includes general SQL capabilities such as scans, > > filters, aggregations, and joins? If that is the long-term goal, would it > > be preferable to keep complex SQL execution in a separate optional SQL > > service or worker process, so that it does not increase the failure domain > > of the REST write Gateway? > > > > > > From: Michael Koepf <[email protected]> > > Date: Thursday, July 30, 2026 at 22:15 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API > > > > Hi Juno, > > > > Thanks for the FIP, great initiative. > > > > A question that comes to my mind: Why not expose metrics via vendor > > neutral OTEL instead of limiting it to Prometheus? > > -- > > Best, > > MKO > > > > > > > On 30.07.2026, at 12:23, Forward Xu <[email protected]> wrote: > > > > > > Thanks junbo, > > > > > > The proposal looks good to me in general. +1 > > > > > > Best, > > > Foward > > > > > > Junbo Wang <[email protected]> 于2026年7月30日周四 11:49写道： > > > > > >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST > > API. > > >> > > >> The proposal is available here: > > >> > > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API > > >> > > >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and > > >> does not conflict with it. Following the earlier suggestion to split the > > >> gateway work into focused FIPs, FIP-49 defines the REST write path and > > DDL > > >> / metadata management in detail, while REST read and the SQL path > > remain in > > >> the scope of FIP-32. > > >> > > >> In short, it proposes a standalone, stateless fluss-gateway process > > (Rust, > > >> axum + tokio, built on fluss-rust) providing: > > >> > > >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row > > >> success/failure reporting > > >> Full database / table / partition DDL > > >> Pluggable client authentication, and a service / user (act-as) identity > > >> mode towards Fluss > > >> Multi-cluster routing, Prometheus metrics, a unified error model > > >> > > >> No Fluss server-side changes are required— the gateway accesses existing > > >> clusters purely as a fluss-rust client. > > >> > > >> Looking forward to your feedback and suggestions. > > >> > > >> > > >> > > >> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x > > >> > > >> > > >> Best regards, > > >> Junbo Wang > > >> > > >> > >
