Hi Flink Devs! I would like to start the discussion about FLIP-599: State Catalog [1]
State and stateful processing has always been one of the most fundamental features of Flink and a major contributor to its success and global adoption. Over the years several apis and methods have been developed to address the need for external access and analytics such as the state processor datastream / java apis, the since deprecated queryable state abstractions and more recently a number of table / SQL api connectors to access state metadata and keyed states in a somewhat limited way. Extending the current capabilities of the state-process-api, this FLIP aims to lift state processing, analytics and observability to a new level by introducing the State Catalog. State Catalog is a Flink SQL Catalog implementation that allows discovering savepoints/checkpoints and mapping their state automatically to SQL tables. The tables are derived for the different operators and their keyed states with schema matching the state structure. Most importantly it supports reading POJO / Avro and other structured and basic type states without the original user classes (dependencies) by relying on Flink's transparent and efficiently structured serializer formats. We have a fully functional prototype implementation developed with Gabor Somogyi that we will be happy to share if the community accepts the proposal! Looking forward to your feedback and suggestions! Gyula [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/438009922/FLIP-599+State+Catalog
