Hey Han! Thanks for checking out the proposal and the thoughtful questions!
>> Checkpoint retention and data availability I feel that introducing external coordination to avoid checkpoint cleanup related issues would be an overkill at this point. Checkpoints are anyways owned by the Flink job but if the user can guarantee they wont be cleaned up (set large retention, or pause the job) then the StateCatalog will work here as well. The primary use-case I think is going to be savepoints initially as it's more straightforward to reason about but the catalog itself can read both savepoints and checkpoints. We can revisit this question for checkpoints as adoption picks up and consider ownership questions in the future. I think the same applies for incremental checkpoints. >> State backend specific format / reader Overall the state processor api uses whatever state backend that you configure. As long as it matches the checkpoint it should work (it uses regular checkpoint restore mechanisms). But it's a very good point to make sure that we detect the state backend format from the state _metadata and actually set that during reading automatically. Cheers Gyula On Wed, Jul 1, 2026 at 1:57 PM Han Yin <[email protected]> wrote: > Hi Gyula, > > Thanks for driving this discussion. The idea of exposing Flink state > through a standard SQL catalog is compelling, and the overall design is > well thought out. > I have a few questions regarding edge cases that the FLIP doesn't seem to > address explicitly: > Checkpoint retention and data availability. Since the catalog operates > independently of the running job, there appears to be no mechanism to "pin" > a checkpoint during a SQL session. This means a checkpoint discovered at > session start could be subsumed by the time a query actually reads its > underlying files. How should we handle this race condition — should there > be some form of reference counting, or is it the user's responsibility to > ensure sufficient retention? > That also brings up the questions about incremental checkpoints, whose > integrity depends on multiple previous checkpoints. > Given that checkpoints and native savepoints use backend-specific storage > formats, does the implementation require a separate reader for each state > backend? > Overall, this is an excellent addition to the Flink ecosystem. Looking > forward to seeing it materialize! > > Best, > Han Yin > > > 2026年6月29日 17:53，Gyula Fóra <[email protected]> 写道： > > > > Hi Flink Devs! > > > > I would like to start the discussion about FLIP-599: State Catalog [1] > > > > State and stateful processing has always been one of the most fundamental > > features of Flink and a major contributor to its success and global > > adoption. > > > > Over the years several apis and methods have been developed to address > the > > need for external access and analytics such as the state processor > > datastream / java apis, the since deprecated queryable state abstractions > > and more recently a number of table / SQL api connectors to access state > > metadata and keyed states in a somewhat limited way. > > > > Extending the current capabilities of the state-process-api, this FLIP > aims > > to lift state processing, analytics and observability to a new level by > > introducing the State Catalog. > > > > State Catalog is a Flink SQL Catalog implementation that allows > discovering > > savepoints/checkpoints and mapping their state automatically to SQL > tables. > > The tables are derived for the different operators and their keyed states > > with schema matching the state structure. Most importantly it supports > > reading POJO / Avro and other structured and basic type states without > the > > original user classes (dependencies) by relying on Flink's transparent > and > > efficiently structured serializer formats. > > > > We have a fully functional prototype implementation developed with Gabor > > Somogyi that we will be happy to share if the community accepts the > > proposal! > > > > Looking forward to your feedback and suggestions! > > > > Gyula > > > > [1] > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/438009922/FLIP-599+State+Catalog > >
