Hello Yang Guo! I went through your new version and you addressed all my points. The proposal now looks neat and the right level of abstraction.
This is a +1 to me. Thank you! On Fri, Aug 14, 2026 at 5:29 PM Lorenzo Affetti < [email protected]> wrote: > Hello Yang Guo, thank you for the FIP! > > I read it together with issue #2362 [1] and its companion design doc [2], > which > proposes sys.lake_tiering_status along similar "system view" lines, and > also > looked at PR #3186 [3], which appears to implement this FIP's framework. A > few > notes, mostly aimed at reconciling the two efforts: > > 1) Coordinator-only serving isn't stated in the FIP text. Tiering status > lives > only in the coordinator's memory, so a view over it needs scanSystemView() > to > be coordinator-only. PR [3] already does this (RpcServiceBase rejects the > call > on TabletService, only CoordinatorService implements it), but the FIP > itself > never says this is a requirement or a supported pattern. #2362 [1][2] > seems to > have been designed without knowledge of this - it defines its own separate > RPC > (ListTieringStatuses on CoordinatorGateway) instead of extending > SystemViewProvider. Worth stating the coordinator-only case explicitly in > the > FIP, and reconciling the two designs so lake_tiering_status becomes one > more > SystemViewProvider rather than a parallel mechanism. > > 2) Small naming inconsistency: the tiering view is called > sys.lake_tiering_status in Motivation and Public Interface, but > lake_tiering_service in "What's Next?". > > 3) The permissions section (Public Interface) reads as a blanket deny for > external reads when the authorizer is disabled. The actual per-type split > (views always readable, persistent tables "depends on the table") only > appears > later, in the guard pseudocode. Worth stating the split up front where > permissions are first introduced. > > 4) SystemViewProvider.scanRows() takes no caller identity, and the guard > model > gives every view unconditional external READ. Fine for > tablet_servers/buckets, > but lake_tiering_status (per #2362 [2]) wants per-row filtering by > DESCRIBE on > the underlying table. Suggest adding a VIEW ResourceType for a coarse > per-view ACL gate, plus passing caller context into scanRows() (or > filtering > RPC-side), so a provider can do row-level filtering where it's needed. > > 5) Proposed Changes fully specifies server-side and Flink-side work, but > never > mentions fluss-client changes, even though the Public Interface examples > depend on them. PR [3] already touches FlussConnection.getTable(), > FlussAdmin.getTableInfo(), and FlussTable > (newScan/newLookup/newAppend/newUpsert). Worth its own explicit subsection. > > 6) Compatibility/Migration Plan is thin for a change that adds two RPCs and > branches core client behavior. It asserts compatibility in one line without > covering the old-client/new-server and new-client/old-server matrix - e.g. > what an old client does with an unrecognized table_kind field, or what > happens > if it attempts a write/lookup against a system view. > > 7) Rejected Alternatives is currently empty. For a design that makes real > choices (view vs. persistent split, a dedicated scan RPC instead of the > standard read path, coordinator-only serving), it'd help to record what > else > was considered. > > Happy to help work through any of this, especially getting #2362 [1][2] > aligned with the framework in PR [3] rather than parallel to it. > > [1] https://github.com/apache/fluss/issues/2362 > [2] > https://docs.google.com/document/d/1eJbRCwzAbeJLA97zQQ0I3JM1jerBXXhq69Dn8r4xWV0/edit?tab=t.0#heading=h.j7m79e2nnoxy > [3] https://github.com/apache/fluss/pull/3186 > > Best, > Lorenzo > > On Wed, Apr 22, 2026 at 3:13 PM Yang Guo <[email protected]> wrote: > >> Hi community, >> >> I'd like to start a discussion on a new feature proposal: introducing >> "system tables" to Fluss.[1] >> >> System tables are tables managed by the cluster itself that reflect >> internal cluster state and bookkeeping information. This concept is now >> well established across streaming and storage systems, like Kafka internal >> topics, PostgreSQL System Catalog[2] and System Views[3], StarRocks >> information_schema[4] and so on. It helps those systems in many aspects >> including metadata management, operations, and maintenance. >> >> This proposal is to introduce system tables into Fluss, like other >> systems already did. It may benefit our project in three key ways: >> 1. Durable storage for system-critical metadata — preparing for more >> features. such as consumer group offsets and capture DDL change events, >> persisted through Fluss's own log storage (LogTablet / KvTablet). >> 2. Operational visibility — exposing cluster runtime state (tablet server >> registrations, bucket ISR, tiering status) via standard SQL query >> interfaces for operators and downstream applications. >> 3. Future ZooKeeper migration — laying the groundwork to move >> ZooKeeper-held metadata into Fluss itself, reducing external dependencies. >> >> For more details, the full draft is available here: FIP-39 >> >> https://cwiki.apache.org/confluence/display/FLUSS/FIP-39%3A+Introduce+System+Tables >> >> I'd really look forward to any of your thoughts and suggestions. Any >> feedback is welcome. >> >> Links: >> [1] FIP-39 Introduce System Tables: >> >> https://cwiki.apache.org/confluence/display/FLUSS/FIP-39%3A+Introduce+System+Tables >> [2] PostgresSQL System Catalog doc: >> https://www.postgresql.org/docs/current/catalogs.html >> [3] PostgresSQL System View doc: >> https://www.postgresql.org/docs/current/views.html >> [4] StarRocks Information Schema doc: >> https://docs.starrocks.io/docs/sql-reference/information_schema/ >> >> Best Regards, >> Yang Guo >> > > > -- > Lorenzo Affetti > Senior Software Engineer @ Flink Team > Ververica <http://www.ververica.com> > -- Lorenzo Affetti Senior Software Engineer @ Flink Team Ververica <http://www.ververica.com>
