Thanks for your clarification. Looking forward to the following FLIPs. Best, Shengkai
Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月23日周一 18:32写道： > Hi Shengkai, > > Thanks for your comments. > > Both state catalog and metadata tables are on the roadmap but not in this > FLIP. > The main approach is to do things incrementally in order to avoid > infinite discussion threads and 10k+ lines pull requests. > > Just to give some teaser, this is how I can roughly imagine upcoming FLIPs > (unordered): > - Expose state metadata on data stream API > - Add metadata support for the SQL connector > - Add non-keyed/operator state data support for the SQL connector > - Add state catalog > > BR, > G > > > On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi. > > > > Thanks Gabor's great FLIP. It is very useful for many cases. But I have > > some suggestions about this FLIP: > > > > 1. It's better to introduce a state Catalog that helps users to translate > > the state information to a table schema. The state catalog uses > > `<operator_name>`.`<state_name>` to find required state and uses state > > descriptor to restore the schema. > > 2. Can the connector provide more metadata information for users? For > > example, the `ttl` field is very useful in our cases. > > > > Best, > > Shengkai > > > > > > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月19日周四 21:41写道： > > > > > Hi Yanquan, > > > > > > Thanks for the question, please see the FLIP content related this. > > > > > > ---BEGIN FLIP CONTENT--- > > > The target of this implementation proposal is to provide SQL support > for > > > keyed states. Other types are going to be covered in further FLIP > > > documents. > > > ---END FLIP CONTENT--- > > > > > > Keyed state data which is defined by users in for example process > > > functions, operator or non-keyed state data is defined by internal > Flink > > > operators for example sources/sinks. > > > In short, we're intended to add this support later on which means that > > > further configuration parameters needs to be added. > > > > > > BR, > > > G > > > > > > > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <decq12y...@gmail.com> > wrote: > > > > > > > Hi, Gabor. Thanks for driving this FLIP and this immediately reminded > > me > > > > of exploring the status of the Source connector like Kafka or > MySQLCDC > > to > > > > understand the progress of reading information. > > > > > > > > In SourceOperator, the state of reader in contained in a ListState > > named > > > > `SourceReaderState`[1], however, the state type of > `SourceReaderState` > > is > > > > `byte[]`, so I am worried that we can't get any understandable > > > information > > > > using this connector. Do we consider supporting this scenario？ > > > > > > > > If we want to support this scenario, perhaps we need to add more > > > > parameters like fields.#.state-version-serializer and > > > > fields.#.state-version to To specify the org.apache.flink.core.io > > > .SimpleVersionedSerializer[2] > > > > and version to be used, And it also requires more complex > > implementation. > > > > > > > > Do I have any misunderstandings? Is this requirement something we > need > > to > > > > support, or will it need support in the future. > > > > > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L109 > > > > [2] > > > > > > > > > > https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java#L50 > > > > > > > > > > > > > > > > > > > > > > > > > 2024年12月18日 17:43，Gabor Somogyi <gabor.g.somo...@gmail.com> 写道： > > > > > > > > > > Hi All, > > > > > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for keyed > > > state > > > > > data [1]. > > > > > Feel free to add your thoughts to make this feature better. > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+state+data > > > > > > > > > > BR, > > > > > G > > > > > > > > > > > > > >
