Hi. Thanks Gabor's great FLIP. It is very useful for many cases. But I have some suggestions about this FLIP:
1. It's better to introduce a state Catalog that helps users to translate the state information to a table schema. The state catalog uses `<operator_name>`.`<state_name>` to find required state and uses state descriptor to restore the schema. 2. Can the connector provide more metadata information for users? For example, the `ttl` field is very useful in our cases. Best, Shengkai Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月19日周四 21:41写道： > Hi Yanquan, > > Thanks for the question, please see the FLIP content related this. > > ---BEGIN FLIP CONTENT--- > The target of this implementation proposal is to provide SQL support for > keyed states. Other types are going to be covered in further FLIP > documents. > ---END FLIP CONTENT--- > > Keyed state data which is defined by users in for example process > functions, operator or non-keyed state data is defined by internal Flink > operators for example sources/sinks. > In short, we're intended to add this support later on which means that > further configuration parameters needs to be added. > > BR, > G > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <decq12y...@gmail.com> wrote: > > > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded me > > of exploring the status of the Source connector like Kafka or MySQLCDC to > > understand the progress of reading information. > > > > In SourceOperator, the state of reader in contained in a ListState named > > `SourceReaderState`[1], however, the state type of `SourceReaderState` is > > `byte[]`, so I am worried that we can't get any understandable > information > > using this connector. Do we consider supporting this scenario？ > > > > If we want to support this scenario, perhaps we need to add more > > parameters like fields.#.state-version-serializer and > > fields.#.state-version to To specify the org.apache.flink.core.io > .SimpleVersionedSerializer[2] > > and version to be used, And it also requires more complex implementation. > > > > Do I have any misunderstandings? Is this requirement something we need to > > support, or will it need support in the future. > > > > > > [1] > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109 > > [2] > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50 > > > > > > > > > > > > > 2024年12月18日 17:43，Gabor Somogyi <gabor.g.somo...@gmail.com> 写道： > > > > > > Hi All, > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed > state > > > data [1]. > > > Feel free to add your thoughts to make this feature better. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data > > > > > > BR, > > > G > > > > >
