Hi Shengkai, Thanks for your comments.
Both state catalog and metadata tables are on the roadmap but not in this FLIP. The main approach is to do things incrementally in order to avoid infinite discussion threads and 10k+ lines pull requests. Just to give some teaser, this is how I can roughly imagine upcoming FLIPs (unordered): - Expose state metadata on data stream API - Add metadata support for the SQL connector - Add non-keyed/operator state data support for the SQL connector - Add state catalog BR, G On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fskm...@gmail.com> wrote: > Hi. > > Thanks Gabor's great FLIP. It is very useful for many cases. But I have > some suggestions about this FLIP: > > 1. It's better to introduce a state Catalog that helps users to translate > the state information to a table schema. The state catalog uses > `<operator_name>`.`<state_name>` to find required state and uses state > descriptor to restore the schema. > 2. Can the connector provide more metadata information for users? For > example, the `ttl` field is very useful in our cases. > > Best, > Shengkai > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月19日周四 21:41写道： > > > Hi Yanquan, > > > > Thanks for the question, please see the FLIP content related this. > > > > ---BEGIN FLIP CONTENT--- > > The target of this implementation proposal is to provide SQL support for > > keyed states. Other types are going to be covered in further FLIP > > documents. > > ---END FLIP CONTENT--- > > > > Keyed state data which is defined by users in for example process > > functions, operator or non-keyed state data is defined by internal Flink > > operators for example sources/sinks. > > In short, we're intended to add this support later on which means that > > further configuration parameters needs to be added. > > > > BR, > > G > > > > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <decq12y...@gmail.com> wrote: > > > > > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded > me > > > of exploring the status of the Source connector like Kafka or MySQLCDC > to > > > understand the progress of reading information. > > > > > > In SourceOperator, the state of reader in contained in a ListState > named > > > `SourceReaderState`[1], however, the state type of `SourceReaderState` > is > > > `byte[]`, so I am worried that we can't get any understandable > > information > > > using this connector. Do we consider supporting this scenario？ > > > > > > If we want to support this scenario, perhaps we need to add more > > > parameters like fields.#.state-version-serializer and > > > fields.#.state-version to To specify the org.apache.flink.core.io > > .SimpleVersionedSerializer[2] > > > and version to be used, And it also requires more complex > implementation. > > > > > > Do I have any misunderstandings? Is this requirement something we need > to > > > support, or will it need support in the future. > > > > > > > > > [1] > > > > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109 > > > [2] > > > > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50 > > > > > > > > > > > > > > > > > > > 2024年12月18日 17:43，Gabor Somogyi <gabor.g.somo...@gmail.com> 写道： > > > > > > > > Hi All, > > > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed > > state > > > > data [1]. > > > > Feel free to add your thoughts to make this feature better. > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data > > > > > > > > BR, > > > > G > > > > > > > > >
