Hi Lorenzo, Thank you for the detailed review. Your comments were very helpful. I’ve updated the FIP-39[1] based on your comments.
The revision clarifies the Coordinator-only execution model for system views, and aligns sys.lake_tiering_status with the common SystemViewProvider framework instead of maintaining a separate RPC path. It also clarifies the permission model, including view-level authorization and caller-aware row filtering. In addition, I documented the required Fluss client changes, expanded the compatibility section, added the alternatives considered, and fixed the naming inconsistencies. One of the core goals of this FIP-39 is to provide a common framework for managing and exposing Fluss’s internal system state, and "lake tiering status" is a representative use case. I agree that #2362 [2] should be integrated into this framework rather than introducing a parallel mechanism. If the integration exposes any limitations or unsuitable assumptions in the framework, please feel free to reach out. I’d be happy to revisit the design and discuss the necessary changes together. Thanks again! Regards, Yang Guo [1] FIP-39: System Tables: https://cwiki.apache.org/confluence/spaces/FLUSS/pages/421956931/FIP-39+System+Tables [2] issue#2362: https://github.com/apache/fluss/issues/2362 On Fri, Aug 14, 2026 at 11:29 PM Lorenzo Affetti via dev < [email protected]> wrote: > Hello Yang Guo, thank you for the FIP! > > I read it together with issue #2362 [1] and its companion design doc [2], > which > proposes sys.lake_tiering_status along similar "system view" lines, and > also > looked at PR #3186 [3], which appears to implement this FIP's framework. A > few > notes, mostly aimed at reconciling the two efforts: > > 1) Coordinator-only serving isn't stated in the FIP text. Tiering status > lives > only in the coordinator's memory, so a view over it needs scanSystemView() > to > be coordinator-only. PR [3] already does this (RpcServiceBase rejects the > call > on TabletService, only CoordinatorService implements it), but the FIP > itself > never says this is a requirement or a supported pattern. #2362 [1][2] seems > to > have been designed without knowledge of this - it defines its own separate > RPC > (ListTieringStatuses on CoordinatorGateway) instead of extending > SystemViewProvider. Worth stating the coordinator-only case explicitly in > the > FIP, and reconciling the two designs so lake_tiering_status becomes one > more > SystemViewProvider rather than a parallel mechanism. > > 2) Small naming inconsistency: the tiering view is called > sys.lake_tiering_status in Motivation and Public Interface, but > lake_tiering_service in "What's Next?". > > 3) The permissions section (Public Interface) reads as a blanket deny for > external reads when the authorizer is disabled. The actual per-type split > (views always readable, persistent tables "depends on the table") only > appears > later, in the guard pseudocode. Worth stating the split up front where > permissions are first introduced. > > 4) SystemViewProvider.scanRows() takes no caller identity, and the guard > model > gives every view unconditional external READ. Fine for > tablet_servers/buckets, > but lake_tiering_status (per #2362 [2]) wants per-row filtering by DESCRIBE > on > the underlying table. Suggest adding a VIEW ResourceType for a coarse > per-view ACL gate, plus passing caller context into scanRows() (or > filtering > RPC-side), so a provider can do row-level filtering where it's needed. > > 5) Proposed Changes fully specifies server-side and Flink-side work, but > never > mentions fluss-client changes, even though the Public Interface examples > depend on them. PR [3] already touches FlussConnection.getTable(), > FlussAdmin.getTableInfo(), and FlussTable > (newScan/newLookup/newAppend/newUpsert). Worth its own explicit subsection. > > 6) Compatibility/Migration Plan is thin for a change that adds two RPCs and > branches core client behavior. It asserts compatibility in one line without > covering the old-client/new-server and new-client/old-server matrix - e.g. > what an old client does with an unrecognized table_kind field, or what > happens > if it attempts a write/lookup against a system view. > > 7) Rejected Alternatives is currently empty. For a design that makes real > choices (view vs. persistent split, a dedicated scan RPC instead of the > standard read path, coordinator-only serving), it'd help to record what > else > was considered. > > Happy to help work through any of this, especially getting #2362 [1][2] > aligned with the framework in PR [3] rather than parallel to it. > > [1] https://github.com/apache/fluss/issues/2362 > [2] > > https://docs.google.com/document/d/1eJbRCwzAbeJLA97zQQ0I3JM1jerBXXhq69Dn8r4xWV0/edit?tab=t.0#heading=h.j7m79e2nnoxy > [3] https://github.com/apache/fluss/pull/3186 > > Best, > Lorenzo > > On Wed, Apr 22, 2026 at 3:13 PM Yang Guo <[email protected]> wrote: > > > Hi community, > > > > I'd like to start a discussion on a new feature proposal: introducing > > "system tables" to Fluss.[1] > > > > System tables are tables managed by the cluster itself that reflect > > internal cluster state and bookkeeping information. This concept is now > > well established across streaming and storage systems, like Kafka > internal > > topics, PostgreSQL System Catalog[2] and System Views[3], StarRocks > > information_schema[4] and so on. It helps those systems in many aspects > > including metadata management, operations, and maintenance. > > > > This proposal is to introduce system tables into Fluss, like other > > systems already did. It may benefit our project in three key ways: > > 1. Durable storage for system-critical metadata — preparing for more > > features. such as consumer group offsets and capture DDL change events, > > persisted through Fluss's own log storage (LogTablet / KvTablet). > > 2. Operational visibility — exposing cluster runtime state (tablet server > > registrations, bucket ISR, tiering status) via standard SQL query > > interfaces for operators and downstream applications. > > 3. Future ZooKeeper migration — laying the groundwork to move > > ZooKeeper-held metadata into Fluss itself, reducing external > dependencies. > > > > For more details, the full draft is available here: FIP-39 > > > > > https://cwiki.apache.org/confluence/display/FLUSS/FIP-39%3A+Introduce+System+Tables > > > > I'd really look forward to any of your thoughts and suggestions. Any > > feedback is welcome. > > > > Links: > > [1] FIP-39 Introduce System Tables: > > > > > https://cwiki.apache.org/confluence/display/FLUSS/FIP-39%3A+Introduce+System+Tables > > [2] PostgresSQL System Catalog doc: > > https://www.postgresql.org/docs/current/catalogs.html > > [3] PostgresSQL System View doc: > > https://www.postgresql.org/docs/current/views.html > > [4] StarRocks Information Schema doc: > > https://docs.starrocks.io/docs/sql-reference/information_schema/ > > > > Best Regards, > > Yang Guo > > > > > -- > Lorenzo Affetti > Senior Software Engineer @ Flink Team > Ververica <http://www.ververica.com> >
