Hi Gabor, - Expose state metadata on data stream API > - Add metadata support for the SQL connector > - Add non-keyed/operator state data support for the SQL connector > - Add state catalog
Thanks for this FLIP. This will be a great addition. Although, most of our use cases are SQL and not DataStream which seems to only benefit when "non-keyed/operator state data support for the SQL connector" is added. Is my understanding correct? Regards Venkata krishnan On Mon, Dec 23, 2024 at 6:59 PM Shengkai Fang <fskm...@gmail.com> wrote: > Thanks for your clarification. Looking forward to the following FLIPs. > > Best, > Shengkai > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月23日周一 18:32写道： > > > Hi Shengkai, > > > > Thanks for your comments. > > > > Both state catalog and metadata tables are on the roadmap but not in this > > FLIP. > > The main approach is to do things incrementally in order to avoid > > infinite discussion threads and 10k+ lines pull requests. > > > > Just to give some teaser, this is how I can roughly imagine upcoming > FLIPs > > (unordered): > > - Expose state metadata on data stream API > > - Add metadata support for the SQL connector > > - Add non-keyed/operator state data support for the SQL connector > > - Add state catalog > > > > BR, > > G > > > > > > On Mon, Dec 23, 2024 at 2:46 AM Shengkai Fang <fskm...@gmail.com> wrote: > > > > > Hi. > > > > > > Thanks Gabor's great FLIP. It is very useful for many cases. But I have > > > some suggestions about this FLIP: > > > > > > 1. It's better to introduce a state Catalog that helps users to > translate > > > the state information to a table schema. The state catalog uses > > > `<operator_name>`.`<state_name>` to find required state and uses state > > > descriptor to restore the schema. > > > 2. Can the connector provide more metadata information for users? For > > > example, the `ttl` field is very useful in our cases. > > > > > > Best, > > > Shengkai > > > > > > > > > > > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2024年12月19日周四 21:41写道： > > > > > > > Hi Yanquan, > > > > > > > > Thanks for the question, please see the FLIP content related this. > > > > > > > > ---BEGIN FLIP CONTENT--- > > > > The target of this implementation proposal is to provide SQL support > > for > > > > keyed states. Other types are going to be covered in further FLIP > > > > documents. > > > > ---END FLIP CONTENT--- > > > > > > > > Keyed state data which is defined by users in for example process > > > > functions, operator or non-keyed state data is defined by internal > > Flink > > > > operators for example sources/sinks. > > > > In short, we're intended to add this support later on which means > that > > > > further configuration parameters needs to be added. > > > > > > > > BR, > > > > G > > > > > > > > > > > > On Thu, Dec 19, 2024 at 2:12 PM Yanquan Lv <decq12y...@gmail.com> > > wrote: > > > > > > > > > Hi, Gabor. Thanks for driving this FLIP and this immediately > reminded > > > me > > > > > of exploring the status of the Source connector like Kafka or > > MySQLCDC > > > to > > > > > understand the progress of reading information. > > > > > > > > > > In SourceOperator, the state of reader in contained in a ListState > > > named > > > > > `SourceReaderState`[1], however, the state type of > > `SourceReaderState` > > > is > > > > > `byte[]`, so I am worried that we can't get any understandable > > > > information > > > > > using this connector. Do we consider supporting this scenario？ > > > > > > > > > > If we want to support this scenario, perhaps we need to add more > > > > > parameters like fields.#.state-version-serializer and > > > > > fields.#.state-version to To specify the org.apache.flink.core.io > > > > .SimpleVersionedSerializer[2] > > > > > and version to be used, And it also requires more complex > > > implementation. > > > > > > > > > > Do I have any misunderstandings? Is this requirement something we > > need > > > to > > > > > support, or will it need support in the future. > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java*L109__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6KTlxdJI$ > > > > > [2] > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/flink/blob/9e38f32889e097f5a9b81f949643a2b037c1035c/flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializer.java*L50__;Iw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6giBFR_Y$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2024年12月18日 17:43，Gabor Somogyi <gabor.g.somo...@gmail.com> 写道： > > > > > > > > > > > > Hi All, > > > > > > > > > > > > I'd like to start a discussion of FLIP-496: SQL connector for > keyed > > > > state > > > > > > data [1]. > > > > > > Feel free to add your thoughts to make this feature better. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-496*3A*SQL*connector*for*keyed*state*data__;JSsrKysrKw!!IKRxdwAv5BmarQ!f-7abNJ6_TcTJTEJmqsVdk7k8XT3k4e-UlBe4cOzZ7-NvL83cup9bQB-HX0VKoaxPlbZyQ7rRrg6980yUcw$ > > > > > > > > > > > > BR, > > > > > > G > > > > > > > > > > > > > > > > > > > >
