Hi all, Given the simplicity, I also +1 for PTF or any other function implementation if PTF is not applicable for this.
I would like to raise a consideration regarding the usage implementation: > Would it be necessary to allow users to utilize the CREATE FUNCTION > statement for registering the PTF? I'd also suggest we make it built-in without registration. Currently, Flink SQL supports letting external systems register modules and > leverage these modules to centrally manage all function definitions. Given > this architectural approach, I’m curious if the plan involves introducing > additional functions in the future. If so, I would advocate for introducing > a dedicated state module to centralize such management. This would empower > users to: I can’t think of any further functions for now, but I'd +1 for a module if it could omit the registration. Best, Zakelly. On Fri, Mar 28, 2025 at 10:25 AM Shengkai Fang <fskm...@gmail.com> wrote: > One more question about the FLIP. > > I think the output schema is definitely a public API to users. If users > use the `CREATE FUNCTION` statement, is it means the class path is also a > public API to users. Alternatively, this is merely an experimental feature > and we don't have any promise about this function. > > Best, > Shengkai > > Shengkai Fang <fskm...@gmail.com> 于2025年3月28日周五 10:20写道： > >> +1 to use PTF. >> >> I would like to raise a consideration regarding the usage implementation: >> Would it be necessary to allow users to utilize the CREATE FUNCTION >> statement for registering the PTF? >> >> Currently, Flink SQL supports letting external systems register modules >> and leverage these modules to centrally manage all function definitions. >> Given this architectural approach, I’m curious if the plan involves >> introducing additional functions in the future. If so, I would advocate for >> introducing a dedicated state module to centralize such management. This >> would empower users to: >> >> 1. Simply execute the LOAD MODULE command to load the required module, and >> 2. Directly invoke read_metadata thereafter. >> >> For more details about the module, please refer to this document[1]. >> >> Best, >> Shengkai >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/modules/ >> >> Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月28日周五 00:26写道： >> >>> Just found out that PTF in batch mode is not supported, plz see the dev >>> mailing about it [1]. >>> >>> [1] https://lists.apache.org/thread/ytm9m1qt4pq2q2gjngfktrn8vrlvkf07 >>> >>> BR, >>> G >>> >>> >>> On Thu, Mar 27, 2025 at 3:38 PM Gabor Somogyi <gabor.g.somo...@gmail.com >>> > >>> wrote: >>> >>> > In the meantime I've just updated the FLIP according to this to be >>> > optimistic 🙂 >>> > >>> > BR, >>> > G >>> > >>> > On Thu, Mar 27, 2025 at 2:15 PM Gabor Somogyi < >>> gabor.g.somo...@gmail.com> >>> > wrote: >>> > >>> >> Considering all the facts I also +1 on PTF. Even if something is >>> missing >>> >> we can add later. >>> >> >>> >> @Zakelly Lan <zakelly....@gmail.com> @Shengkai Fang are you also on >>> the >>> >> same page or have something to add? >>> >> >>> >> BR, >>> >> G >>> >> >>> >> >>> >> On Thu, Mar 27, 2025 at 1:50 PM Lincoln Lee <lincoln.8...@gmail.com> >>> >> wrote: >>> >> >>> >>> +1 for PTF >>> >>> >>> >>> > Is it possible to describe such function to see the column >>> names/types? >>> >>> >>> >>> Although Flink SQL does not directly support this feature, users can >>> >>> achieve >>> >>> similar results with the help of `explain` syntax, e.g. >>> >>> 'explain select * from read_state_metadata(...)' >>> >>> >>> >>> >>> >>> Best, >>> >>> Lincoln Lee >>> >>> >>> >>> >>> >>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月27日周四 20:41写道： >>> >>> >>> >>> > Hey! >>> >>> > >>> >>> > I think the PTF approach strikes a great balance in simplicity and >>> the >>> >>> > capabilities that we get out of it. >>> >>> > >>> >>> > I think this could be a completely viable alternative to the >>> dedicated >>> >>> > connector, +1. >>> >>> > >>> >>> > Cheers, >>> >>> > Gyula >>> >>> > >>> >>> > On Thu, Mar 27, 2025 at 10:37 AM Shengkai Fang <fskm...@gmail.com> >>> >>> wrote: >>> >>> > >>> >>> > > Hi, Gabor. >>> >>> > > >>> >>> > > > Do I understand correctly that this is 2.x only feature and we >>> >>> can't >>> >>> > > backport it to 1.x line >>> >>> > > >>> >>> > > Yes. PTF is only supported in 2.x verison. >>> >>> > > >>> >>> > > > Is it possible to describe such function to see the column >>> >>> names/types? >>> >>> > > >>> >>> > > Flink SQL doesn't support this feature, but postgres[2] or >>> mysql[1] >>> >>> has >>> >>> > > similar feature. >>> >>> > > >>> >>> > > [1] >>> >>> https://dev.mysql.com/doc/refman/8.4/en/show-create-procedure.html >>> >>> > > [2] >>> >>> > > >>> >>> > > >>> >>> > >>> >>> >>> https://stackoverflow.com/questions/6898453/show-the-code-of-a-function-procedure-and-trigger-in-postgresql >>> >>> > > >>> >>> > > Best, >>> >>> > > Shengkai >>> >>> > > >>> >>> > > >>> >>> > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月27日周四 16:25写道： >>> >>> > > >>> >>> > > > Hi Shengkai, >>> >>> > > > >>> >>> > > > Thanks for your effort with the example, this looks promising. >>> >>> > > > I like the fact that users wouldn't need to sweat with complex >>> >>> create >>> >>> > > table >>> >>> > > > statements. >>> >>> > > > >>> >>> > > > Couple of questions: >>> >>> > > > * Do I understand correctly that this is 2.x only feature and >>> we >>> >>> can't >>> >>> > > > backport it to 1.x line? >>> >>> > > > I'm not intended to do any backport, just would like to know >>> the >>> >>> > > technical >>> >>> > > > constraints. >>> >>> > > > * Is it possible to describe such function to see the column >>> >>> > names/types? >>> >>> > > > >>> >>> > > > BR, >>> >>> > > > G >>> >>> > > > >>> >>> > > > >>> >>> > > > On Thu, Mar 27, 2025 at 3:17 AM Shengkai Fang < >>> fskm...@gmail.com> >>> >>> > wrote: >>> >>> > > > >>> >>> > > > > Many thanks for your reminder, Leonard. Here's the link I >>> >>> > mentioned[1]. >>> >>> > > > > >>> >>> > > > > Best, >>> >>> > > > > Shengkai >>> >>> > > > > >>> >>> > > > > [1] https://github.com/apache/flink/pull/26358 >>> >>> > > > > >>> >>> > > > > Leonard Xu <xbjt...@gmail.com> 于2025年3月27日周四 10:05写道： >>> >>> > > > > >>> >>> > > > > > Your link is broken, Shengkai >>> >>> > > > > > >>> >>> > > > > > Best, >>> >>> > > > > > Leonard >>> >>> > > > > > >>> >>> > > > > > > 2025年3月27日 10:01，Shengkai Fang <fskm...@gmail.com> 写道： >>> >>> > > > > > > >>> >>> > > > > > > Hi, All. >>> >>> > > > > > > >>> >>> > > > > > > I write a simple demo to illustrate my idea. Hope this >>> helps. >>> >>> > > > > > > >>> >>> > > > > > > Best, >>> >>> > > > > > > Shengkai >>> >>> > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1 >>> >>> > > > > > > >>> >>> > > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月26日周三 >>> >>> 15:54写道： >>> >>> > > > > > > >>> >>> > > > > > >>> I'm fine with a seperate SQL connector for metadata, so >>> >>> maybe >>> >>> > we >>> >>> > > > > could >>> >>> > > > > > >> update the FLIP about our discussion? >>> >>> > > > > > >> >>> >>> > > > > > >> Sorry, I've forgotten this part. Yeah, no matter we >>> choose >>> >>> I'm >>> >>> > > going >>> >>> > > > > to >>> >>> > > > > > >> update the FLIP. >>> >>> > > > > > >> >>> >>> > > > > > >> G >>> >>> > > > > > >> >>> >>> > > > > > >> >>> >>> > > > > > >> On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi < >>> >>> > > > > > gabor.g.somo...@gmail.com> >>> >>> > > > > > >> wrote: >>> >>> > > > > > >> >>> >>> > > > > > >>> Hi All, >>> >>> > > > > > >>> >>> >>> > > > > > >>> I've also lack of the knowledge of PTF so I've read >>> just >>> >>> the >>> >>> > > > > motivation >>> >>> > > > > > >>> part: >>> >>> > > > > > >>> >>> >>> > > > > > >>> "The SQL 2016 standard introduced a way of defining >>> custom >>> >>> SQL >>> >>> > > > > > operators >>> >>> > > > > > >>> defined by ISO/IEC 19075-7:2021 (Part 7: Polymorphic >>> table >>> >>> > > > > functions). >>> >>> > > > > > >>> ~200 pages define how this new kind of function can >>> >>> consume and >>> >>> > > > > produce >>> >>> > > > > > >>> tables with various execution properties. >>> >>> > > > > > >>> Unfortunately, this part of the standard is not >>> publicly >>> >>> > > > available." >>> >>> > > > > > >>> >>> >>> > > > > > >>> Of course we can take a look at some examples but do we >>> >>> really >>> >>> > > want >>> >>> > > > > to >>> >>> > > > > > >>> expose state data with this construct >>> >>> > > > > > >>> which is described in ~200 pages and part of the >>> standard >>> >>> is >>> >>> > not >>> >>> > > > > > publicly >>> >>> > > > > > >>> available? 🙂 >>> >>> > > > > > >>> I mean the dataset is couple of rows and the use-case >>> is >>> >>> join >>> >>> > > with >>> >>> > > > > > >> another >>> >>> > > > > > >>> table like with state data. >>> >>> > > > > > >>> If somebody can give advantages I would buy that but >>> from >>> >>> my >>> >>> > > > limited >>> >>> > > > > > >>> understanding this would be an overkill here. >>> >>> > > > > > >>> >>> >>> > > > > > >>> BR, >>> >>> > > > > > >>> G >>> >>> > > > > > >>> >>> >>> > > > > > >>> >>> >>> > > > > > >>> On Wed, Mar 26, 2025 at 8:28 AM Gyula Fóra < >>> >>> > gyula.f...@gmail.com >>> >>> > > > >>> >>> > > > > > wrote: >>> >>> > > > > > >>> >>> >>> > > > > > >>>> Hi Zakelly , Shengkai! >>> >>> > > > > > >>>> >>> >>> > > > > > >>>> I don't know too much about PTFs, it would be >>> interesting >>> >>> to >>> >>> > see >>> >>> > > > how >>> >>> > > > > > the >>> >>> > > > > > >>>> usage would look in practice. >>> >>> > > > > > >>>> >>> >>> > > > > > >>>> Do you have some mockup/example in mind how the PTF >>> would >>> >>> look >>> >>> > > for >>> >>> > > > > > >> example >>> >>> > > > > > >>>> when want to: >>> >>> > > > > > >>>> - Simply display/aggregate whats in the metadata >>> >>> > > > > > >>>> - Join keyed state with some metadata columns >>> >>> > > > > > >>>> >>> >>> > > > > > >>>> Thanks >>> >>> > > > > > >>>> Gyula >>> >>> > > > > > >>>> >>> >>> > > > > > >>>> On Wed, Mar 26, 2025 at 7:33 AM Zakelly Lan < >>> >>> > > > zakelly....@gmail.com> >>> >>> > > > > > >>>> wrote: >>> >>> > > > > > >>>> >>> >>> > > > > > >>>>> Hi everyone, >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>> I'm fine with a seperate SQL connector for metadata, >>> so >>> >>> maybe >>> >>> > > we >>> >>> > > > > > could >>> >>> > > > > > >>>>> update the FLIP about our discussion? And Shengkai >>> >>> provides a >>> >>> > > PTF >>> >>> > > > > > >>>>> implementation, does that also meet the requirement? >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>> Best, >>> >>> > > > > > >>>>> Zakelly >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>> On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi < >>> >>> > > > > > >>>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>> wrote: >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>>> Hi All, >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>> @Zakelly: Gyula summarised it correctly what I >>> meant so >>> >>> > please >>> >>> > > > > treat >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>> content as mine. >>> >>> > > > > > >>>>>> As an addition I'm not against to add CLI at all, >>> I'm >>> >>> just >>> >>> > > > stating >>> >>> > > > > > >>>> that >>> >>> > > > > > >>>>> in >>> >>> > > > > > >>>>>> some cases like this, users would like to have >>> >>> > > > > > >>>>>> a self-serving solution where they can provide SQL >>> >>> > statements >>> >>> > > > > which >>> >>> > > > > > >>>> can >>> >>> > > > > > >>>>>> trigger alerts automatically. >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>> My personal opinion is that CLI would be beneficial >>> for >>> >>> > > several >>> >>> > > > > > >>>> cases. A >>> >>> > > > > > >>>>>> good example is when users want to restart job >>> >>> > > > > > >>>>>> from specific Kafka offsets which are persisted in a >>> >>> > > savepoint. >>> >>> > > > > For >>> >>> > > > > > >>>> such >>> >>> > > > > > >>>>>> scenario users are more than happy since they >>> >>> > > > > > >>>>>> expect manual intervention with full control. So >>> all in >>> >>> all >>> >>> > > one >>> >>> > > > > can >>> >>> > > > > > >>>> count >>> >>> > > > > > >>>>>> on my +1 when CLI FLIP would come up... >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>> BR, >>> >>> > > > > > >>>>>> G >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>> On Thu, Mar 20, 2025 at 8:20 AM Gyula Fóra < >>> >>> > > > gyula.f...@gmail.com> >>> >>> > > > > > >>>> wrote: >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>>>> Hi! >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>> @Zakelly Lan <zakelly....@gmail.com> >>> >>> > > > > > >>>>>>> I think what Gabor means is that users want to have >>> >>> > > predefined >>> >>> > > > > SQL >>> >>> > > > > > >>>>> scripts >>> >>> > > > > > >>>>>>> to perform state analysis tasks to debug/identify >>> >>> problems. >>> >>> > > > > > >>>>>>> Such as write a SQL script that joins the metadata >>> >>> table >>> >>> > with >>> >>> > > > the >>> >>> > > > > > >>>> state >>> >>> > > > > > >>>>>>> and >>> >>> > > > > > >>>>>>> do some analytics on it. >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>> If we have a meta table then the SQL script that >>> can do >>> >>> > this >>> >>> > > is >>> >>> > > > > > >> fixed >>> >>> > > > > > >>>>> and >>> >>> > > > > > >>>>>>> users can trigger this on demand by simply >>> providing a >>> >>> new >>> >>> > > > > > >> savepoint >>> >>> > > > > > >>>>> path. >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>> If we have a different mechanism to extract >>> metadata >>> >>> that >>> >>> > is >>> >>> > > > not >>> >>> > > > > > >> SQL >>> >>> > > > > > >>>>>>> native >>> >>> > > > > > >>>>>>> then manual steps need to be executed and a custom >>> SQL >>> >>> > script >>> >>> > > > > would >>> >>> > > > > > >>>> need >>> >>> > > > > > >>>>>>> to >>> >>> > > > > > >>>>>>> be written that adds the manually extracted >>> metadata >>> >>> into >>> >>> > the >>> >>> > > > > > >> script. >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>> Cheers, >>> >>> > > > > > >>>>>>> Gyula >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>> On Thu, Mar 20, 2025 at 4:32 AM Zakelly Lan < >>> >>> > > > > zakelly....@gmail.com >>> >>> > > > > > >>> >>> >>> > > > > > >>>>>>> wrote: >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>>>> Hi all, >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> Thanks for your answers! Getting everyone aligned >>> on >>> >>> this >>> >>> > > > topic >>> >>> > > > > > >> is >>> >>> > > > > > >>>>>>>> challenging, but it’s definitely worth the effort >>> >>> since it >>> >>> > > > will >>> >>> > > > > > >>>> help >>> >>> > > > > > >>>>>>>> streamline things moving forward. >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> @Gabor are you saying that users are using some >>> >>> scripts to >>> >>> > > > > define >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>> SQL >>> >>> > > > > > >>>>>>>> metadata connector and get the information, >>> right? If >>> >>> so, >>> >>> > > > would >>> >>> > > > > a >>> >>> > > > > > >>>> CLI >>> >>> > > > > > >>>>>>> tool >>> >>> > > > > > >>>>>>>> be more convenient? It's easy to invoke and can >>> get >>> >>> the >>> >>> > > result >>> >>> > > > > > >>>>> swiftly. >>> >>> > > > > > >>>>>>> And >>> >>> > > > > > >>>>>>>> there should be some other systems to track the >>> >>> checkpoint >>> >>> > > > > > >> lineage >>> >>> > > > > > >>>> and >>> >>> > > > > > >>>>>>>> analyze if there are outliers in metadata (e.g. >>> state >>> >>> size >>> >>> > > of >>> >>> > > > > one >>> >>> > > > > > >>>>>>> operator) >>> >>> > > > > > >>>>>>>> right? Well, maybe I missed something so please >>> >>> correct me >>> >>> > > if >>> >>> > > > > I'm >>> >>> > > > > > >>>>> wrong. >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> I think the overall vision in Flink SQL is to >>> provide >>> >>> a >>> >>> > SQL >>> >>> > > > > > >> native >>> >>> > > > > > >>>>>>>>> environment where we can serve complex use-cases >>> >>> like you >>> >>> > > > would >>> >>> > > > > > >>>>> expect >>> >>> > > > > > >>>>>>>> in a >>> >>> > > > > > >>>>>>>>> regular database. >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> @Gyula Well, this is a good point. From the >>> >>> perspective of >>> >>> > > > > > >>>>> comprehensive >>> >>> > > > > > >>>>>>>> SQL experience, I'd +1 for treating metadata as >>> data. >>> >>> > > > Although I >>> >>> > > > > > >>>> doubt >>> >>> > > > > > >>>>>>> if >>> >>> > > > > > >>>>>>>> there is a need for processing metadata, I won't >>> be >>> >>> > against >>> >>> > > a >>> >>> > > > > > >>>> separate >>> >>> > > > > > >>>>>>>> connector. >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> Regarding the CLI tool, I still think it’s worth >>> >>> > > implementing. >>> >>> > > > > > >>>> Such a >>> >>> > > > > > >>>>>>> tool >>> >>> > > > > > >>>>>>>> could provide savepoint information before >>> resuming >>> >>> from a >>> >>> > > > > > >>>> savepoint, >>> >>> > > > > > >>>>>>> which >>> >>> > > > > > >>>>>>>> would enhance the user experience in CLI-based >>> >>> workflows. >>> >>> > It >>> >>> > > > > > >> would >>> >>> > > > > > >>>> be >>> >>> > > > > > >>>>>>> good >>> >>> > > > > > >>>>>>>> if someone could implement this feature. We >>> shouldn’t >>> >>> > worry >>> >>> > > > > about >>> >>> > > > > > >>>>>>> whether >>> >>> > > > > > >>>>>>>> this tool might be retired in the future. >>> Regardless >>> >>> of >>> >>> > the >>> >>> > > > > > >>>> SQL-based >>> >>> > > > > > >>>>>>>> solution we eventually adopt, this capability will >>> >>> remain >>> >>> > > > > > >> essential >>> >>> > > > > > >>>>> for >>> >>> > > > > > >>>>>>> CLI >>> >>> > > > > > >>>>>>>> users. This is another topic. >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> Best, >>> >>> > > > > > >>>>>>>> Zakelly >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>> On Thu, Mar 20, 2025 at 10:37 AM Shengkai Fang < >>> >>> > > > > > >> fskm...@gmail.com> >>> >>> > > > > > >>>>>>> wrote: >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>>>> Hi. >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> After reading the doc[1], I think Spark provides >>> a >>> >>> > function >>> >>> > > > for >>> >>> > > > > > >>>>> users >>> >>> > > > > > >>>>>>> to >>> >>> > > > > > >>>>>>>>> consume the metadata from the savepoint. In >>> Flink >>> >>> SQL, >>> >>> > > > similar >>> >>> > > > > > >>>>>>>>> functionality is implemented through Polymorphic >>> >>> Table >>> >>> > > > > > >> Functions >>> >>> > > > > > >>>>>>> (PTF) as >>> >>> > > > > > >>>>>>>>> proposed in FLIP-440[2]. Below is a code >>> example[3] >>> >>> > > > > > >> illustrating >>> >>> > > > > > >>>>> this >>> >>> > > > > > >>>>>>>>> concept: >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> ``` >>> >>> > > > > > >>>>>>>>> public static class ScalarArgsFunction extends >>> >>> > > > > > >>>>>>>>> TestProcessTableFunctionBase { >>> >>> > > > > > >>>>>>>>> public void eval(Integer i, Boolean b) { >>> >>> > > > > > >>>>>>>>> collectObjects(i, b); >>> >>> > > > > > >>>>>>>>> } >>> >>> > > > > > >>>>>>>>> } >>> >>> > > > > > >>>>>>>>> ``` >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> ``` >>> >>> > > > > > >>>>>>>>> INSERT INTO sink SELECT * FROM f(i => 42, b => >>> >>> > CAST('TRUE' >>> >>> > > AS >>> >>> > > > > > >>>>>>> BOOLEAN)) >>> >>> > > > > > >>>>>>>>> `` >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> So we can add a builtin function named >>> >>> > > `read_state_metadata` >>> >>> > > > to >>> >>> > > > > > >>>> read >>> >>> > > > > > >>>>>>>>> savepoint data. >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://docs.databricks.com/aws/en/structured-streaming/read-state?language=SQL >>> >>> > > > > > >>>>>>>>> [2] >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=298781093 >>> >>> > > > > > >>>>>>>>> [3] >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java#L140 >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月19日周三 >>> >>> 18:37写道： >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>>>> Hi All! >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> Thank you for the answers and concerns from >>> >>> everyone. >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> On the CLI vs State Metadata Connector/Table >>> >>> question I >>> >>> > > > would >>> >>> > > > > > >>>> also >>> >>> > > > > > >>>>>>> like >>> >>> > > > > > >>>>>>>>> to >>> >>> > > > > > >>>>>>>>>> step back a little and look at the bigger >>> picture. >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> I think the overall vision in Flink SQL is to >>> >>> provide a >>> >>> > > SQL >>> >>> > > > > > >>>> native >>> >>> > > > > > >>>>>>>>>> environment where we can serve complex use-cases >>> >>> like >>> >>> > you >>> >>> > > > > > >> would >>> >>> > > > > > >>>>>>> expect >>> >>> > > > > > >>>>>>>>> in a >>> >>> > > > > > >>>>>>>>>> regular database. >>> >>> > > > > > >>>>>>>>>> Most features, developments in the recent years >>> have >>> >>> > gone >>> >>> > > > > > >> this >>> >>> > > > > > >>>>> way. >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> The State Metadata Table would be a natural and >>> >>> > > > > > >> straightforward >>> >>> > > > > > >>>>> fit >>> >>> > > > > > >>>>>>>> here. >>> >>> > > > > > >>>>>>>>>> So from my side, +1 for that. >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> However I could understand if we are not ready >>> to >>> >>> add a >>> >>> > > new >>> >>> > > > > > >>>>>>>>>> connector/format due to maintenance concerns >>> (and in >>> >>> > > general >>> >>> > > > > > >>>>> concern >>> >>> > > > > > >>>>>>>>> about >>> >>> > > > > > >>>>>>>>>> the design). >>> >>> > > > > > >>>>>>>>>> If that's the issue then we should spend more >>> time >>> >>> on >>> >>> > the >>> >>> > > > > > >>>> design >>> >>> > > > > > >>>>> to >>> >>> > > > > > >>>>>>> get >>> >>> > > > > > >>>>>>>>>> comfortable with the approach and seek feedback >>> >>> from the >>> >>> > > > > > >> wider >>> >>> > > > > > >>>>>>>> community >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> I am -1 for the CLI/tooling approach as that >>> will >>> >>> not >>> >>> > > > provide >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>>> featureset we are looking for that is not >>> already >>> >>> > covered >>> >>> > > by >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>> Java >>> >>> > > > > > >>>>>>>>>> connector. And that approach would come with the >>> >>> same >>> >>> > > > > > >>>> maintenance >>> >>> > > > > > >>>>>>>>>> implications. >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> Cheers >>> >>> > > > > > >>>>>>>>>> Gyula >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> On Wed, Mar 19, 2025 at 11:24 AM Gabor Somogyi < >>> >>> > > > > > >>>>>>>>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> Hi Zaklely, Shengkai >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> Several topics are going on so adding gist >>> answers >>> >>> to >>> >>> > > them. >>> >>> > > > > > >>>> When >>> >>> > > > > > >>>>>>> some >>> >>> > > > > > >>>>>>>>>> topic >>> >>> > > > > > >>>>>>>>>>> is not touched please highlight it. >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> @Shengkai: I've read through all the previous >>> FLIPs >>> >>> > > related >>> >>> > > > > > >>>>>>> catalogs >>> >>> > > > > > >>>>>>>>> and >>> >>> > > > > > >>>>>>>>>> if >>> >>> > > > > > >>>>>>>>>>> we would like to keep the concepts there >>> >>> > > > > > >>>>>>>>>>> then one-to-one mapping relationship between >>> >>> savepoint >>> >>> > > and >>> >>> > > > > > >>>>> catalog >>> >>> > > > > > >>>>>>>> is a >>> >>> > > > > > >>>>>>>>>>> reasonable direction. In short I'm happy that >>> >>> > > > > > >>>>>>>>>>> you've highlighted this and agree as a whole. >>> I've >>> >>> > > written >>> >>> > > > > > >> it >>> >>> > > > > > >>>>> down >>> >>> > > > > > >>>>>>>>>>> previously, just want to double confirm that >>> state >>> >>> > > catalog >>> >>> > > > > > >> is >>> >>> > > > > > >>>>>>>>>>> essential and planned. When we reach this point >>> >>> then >>> >>> > your >>> >>> > > > > > >>>> input >>> >>> > > > > > >>>>> is >>> >>> > > > > > >>>>>>>> more >>> >>> > > > > > >>>>>>>>>>> than welcome. >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> @Zakelly: We've tried the CLI and separate >>> library >>> >>> > > > > > >> approaches >>> >>> > > > > > >>>>> with >>> >>> > > > > > >>>>>>>>> users >>> >>> > > > > > >>>>>>>>>>> already and these are not something which is >>> >>> welcome >>> >>> > > > > > >> because >>> >>> > > > > > >>>> of >>> >>> > > > > > >>>>>>> the >>> >>> > > > > > >>>>>>>>>>> following: >>> >>> > > > > > >>>>>>>>>>> * Users want to have automated tasks and not >>> manual >>> >>> > > > > > >>>> CLI/library >>> >>> > > > > > >>>>>>>> output >>> >>> > > > > > >>>>>>>>>>> parsing. This can be hacked around but our >>> >>> experience >>> >>> > is >>> >>> > > > > > >>>>> negative >>> >>> > > > > > >>>>>>> on >>> >>> > > > > > >>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>> because it's just brittle. >>> >>> > > > > > >>>>>>>>>>> * From development perspective It's way much >>> bigger >>> >>> > > effort >>> >>> > > > > > >>>> than >>> >>> > > > > > >>>>> a >>> >>> > > > > > >>>>>>>>>> connector >>> >>> > > > > > >>>>>>>>>>> (hard to test, packaging/version handling is >>> and >>> >>> extra >>> >>> > > > > > >> layer >>> >>> > > > > > >>>> of >>> >>> > > > > > >>>>>>>>>> complexity, >>> >>> > > > > > >>>>>>>>>>> external FS authentication is pain for users, >>> >>> expecting >>> >>> > > > > > >> them >>> >>> > > > > > >>>> to >>> >>> > > > > > >>>>>>>>> download >>> >>> > > > > > >>>>>>>>>>> savepoints also) >>> >>> > > > > > >>>>>>>>>>> * Purely personal opinion but if we would find >>> >>> better >>> >>> > > ways >>> >>> > > > > > >>>> later >>> >>> > > > > > >>>>>>> then >>> >>> > > > > > >>>>>>>>>>> retire a CLI is not more lightweight than >>> retire a >>> >>> > > > > > >> connector >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> It would be great if you give some examples >>> on how >>> >>> > user >>> >>> > > > > > >>>> could >>> >>> > > > > > >>>>>>>>> leverage >>> >>> > > > > > >>>>>>>>>>> the separate connector to process the metadata. >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> The most simplest cases: >>> >>> > > > > > >>>>>>>>>>> * give me the overgroving state uids >>> >>> > > > > > >>>>>>>>>>> * give me the not known (new or renamed) state >>> uids >>> >>> > > > > > >>>>>>>>>>> * give me the state uids where state size >>> >>> drastically >>> >>> > > > > > >> dropped >>> >>> > > > > > >>>>>>> compare >>> >>> > > > > > >>>>>>>>> to >>> >>> > > > > > >>>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>> previous savepoint (accidental state loss) >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> Since it was mentioned: as a general offtopic >>> >>> teaser, >>> >>> > > yeah >>> >>> > > > > > >> it >>> >>> > > > > > >>>>>>> would >>> >>> > > > > > >>>>>>>> be >>> >>> > > > > > >>>>>>>>>> good >>> >>> > > > > > >>>>>>>>>>> to have some sort of checkpoint/savepoint >>> lineage >>> >>> or >>> >>> > > > > > >> however >>> >>> > > > > > >>>> we >>> >>> > > > > > >>>>>>> call >>> >>> > > > > > >>>>>>>>> it. >>> >>> > > > > > >>>>>>>>>>> Since we've not yet reached this point there >>> are no >>> >>> > > > > > >> technical >>> >>> > > > > > >>>>>>>> details, >>> >>> > > > > > >>>>>>>>>> it's >>> >>> > > > > > >>>>>>>>>>> more like a vision. It's a common pattern that >>> >>> > > > > > >>>>>>>>>>> jobs are physically running but somehow the >>> state >>> >>> > > > > > >> processing >>> >>> > > > > > >>>> is >>> >>> > > > > > >>>>>>> stuck >>> >>> > > > > > >>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>> it would be good to add some way to find it out >>> >>> > > > > > >>>> automatically. >>> >>> > > > > > >>>>>>>>>>> The important saying here is automation and not >>> >>> manual >>> >>> > > > > > >>>>> evaluation >>> >>> > > > > > >>>>>>>> since >>> >>> > > > > > >>>>>>>>>>> handling 10k+ jobs is just not allowing that. >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> On Wed, Mar 19, 2025 at 6:46 AM Shengkai Fang < >>> >>> > > > > > >>>>> fskm...@gmail.com> >>> >>> > > > > > >>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> Hi, All. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> About State Catalog, I want to share more >>> thoughts >>> >>> > about >>> >>> > > > > > >>>> this. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> In the initial design concept, I understood >>> that a >>> >>> > > > > > >>>> savepoint >>> >>> > > > > > >>>>>>> and a >>> >>> > > > > > >>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>> catalog have a one-to-one mapping >>> relationship. >>> >>> Each >>> >>> > > > > > >>>> operator >>> >>> > > > > > >>>>>>>>>> corresponds >>> >>> > > > > > >>>>>>>>>>>> to a database, and the state of each operator >>> is >>> >>> > > > > > >>>> represented >>> >>> > > > > > >>>>> as >>> >>> > > > > > >>>>>>>>>>> individual >>> >>> > > > > > >>>>>>>>>>>> tables. The rationale behind this design is: >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> *State Diversity*: An operator may involve >>> >>> multiple >>> >>> > > types >>> >>> > > > > > >>>> of >>> >>> > > > > > >>>>>>>> states. >>> >>> > > > > > >>>>>>>>>> For >>> >>> > > > > > >>>>>>>>>>>> example, in our VVR design, a "multi-join" >>> >>> operator >>> >>> > uses >>> >>> > > > > > >>>> keyed >>> >>> > > > > > >>>>>>>> states >>> >>> > > > > > >>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>> two input streams and a broadcast state for >>> the >>> >>> third >>> >>> > > > > > >>>> stream. >>> >>> > > > > > >>>>>>> This >>> >>> > > > > > >>>>>>>>>> makes >>> >>> > > > > > >>>>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>> challenging to represent all states of an >>> operator >>> >>> > > > > > >> within a >>> >>> > > > > > >>>>>>> single >>> >>> > > > > > >>>>>>>>>> table. >>> >>> > > > > > >>>>>>>>>>>> *Scalability*: Internally, an operator might >>> have >>> >>> > > > > > >> multiple >>> >>> > > > > > >>>>> keyed >>> >>> > > > > > >>>>>>>>> states >>> >>> > > > > > >>>>>>>>>>>> (e.g., value state and list state). However, >>> large >>> >>> > list >>> >>> > > > > > >>>> states >>> >>> > > > > > >>>>>>> may >>> >>> > > > > > >>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>> fit >>> >>> > > > > > >>>>>>>>>>>> entirely in memory. To address this, we >>> recommend >>> >>> > > > > > >>>> implementing >>> >>> > > > > > >>>>>>> each >>> >>> > > > > > >>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>> as a separate table. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> To resolve the loosely coupled relationships >>> >>> between >>> >>> > > > > > >>>> operator >>> >>> > > > > > >>>>>>>> states, >>> >>> > > > > > >>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>> propose embedding predefined views within the >>> >>> catalog. >>> >>> > > > > > >>>> These >>> >>> > > > > > >>>>>>> views >>> >>> > > > > > >>>>>>>>>>> simplify >>> >>> > > > > > >>>>>>>>>>>> user understanding of operator >>> implementations and >>> >>> > > > > > >> provide >>> >>> > > > > > >>>> a >>> >>> > > > > > >>>>>>> more >>> >>> > > > > > >>>>>>>>>>> intuitive >>> >>> > > > > > >>>>>>>>>>>> perspective. For instance, a join operator may >>> >>> have >>> >>> > > > > > >>>> multiple >>> >>> > > > > > >>>>>>> state >>> >>> > > > > > >>>>>>>>>>>> implementations (depending on whether the >>> join key >>> >>> > > > > > >> includes >>> >>> > > > > > >>>>>>> unique >>> >>> > > > > > >>>>>>>>>>>> attributes), but users primarily care about >>> the >>> >>> data >>> >>> > > > > > >>>>> associated >>> >>> > > > > > >>>>>>>> with >>> >>> > > > > > >>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>>> specific join key across input streams. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> Returning to the one-to-one mapping between >>> >>> savepoints >>> >>> > > > > > >> and >>> >>> > > > > > >>>>>>>> catalogs, >>> >>> > > > > > >>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>> aim >>> >>> > > > > > >>>>>>>>>>>> to manage multiple user state catalogs >>> through a >>> >>> > catalog >>> >>> > > > > > >>>>> store. >>> >>> > > > > > >>>>>>>> When >>> >>> > > > > > >>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>> user >>> >>> > > > > > >>>>>>>>>>>> triggers a savepoint for a job on the >>> platform: >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> 1. The platform sends a REST request to the >>> >>> > JobManager. >>> >>> > > > > > >>>>>>>>>>>> 2. Simultaneously, it registers a new state >>> >>> catalog in >>> >>> > > > > > >> the >>> >>> > > > > > >>>>>>> catalog >>> >>> > > > > > >>>>>>>>>> store, >>> >>> > > > > > >>>>>>>>>>>> enabling immediate analysis of state data on >>> the >>> >>> > > > > > >> platform. >>> >>> > > > > > >>>>>>>>>>>> 3. Deleting a savepoint would also trigger the >>> >>> removal >>> >>> > > of >>> >>> > > > > > >>>> its >>> >>> > > > > > >>>>>>>>>> associated >>> >>> > > > > > >>>>>>>>>>>> catalog. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> This vision assumes that states are >>> >>> self-describing or >>> >>> > > > > > >>>> that a >>> >>> > > > > > >>>>>>> state >>> >>> > > > > > >>>>>>>>>>>> metaservice is introduced to analyze savepoint >>> >>> > > > > > >> structures. >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> How can users create logic to identify >>> >>> differences >>> >>> > > > > > >>>> between >>> >>> > > > > > >>>>>>>> multiple >>> >>> > > > > > >>>>>>>>>>>> savepoints? >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> Since savepoints and state catalogs are >>> one-to-one >>> >>> > > > > > >> mapped, >>> >>> > > > > > >>>>> users >>> >>> > > > > > >>>>>>>> can >>> >>> > > > > > >>>>>>>>>>> query >>> >>> > > > > > >>>>>>>>>>>> metadata via their respective catalogs. For >>> >>> example: >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> 1. >>> >>> > > > > > >>>>> >>> >>> `savepoint-${id}`.`system`.`metadata_table`.`<operator-name>` >>> >>> > > > > > >>>>>>>>>> provides >>> >>> > > > > > >>>>>>>>>>>> operator-specific metadata (e.g., state size, >>> >>> type). >>> >>> > > > > > >>>>>>>>>>>> 2. Comparing metadata tables (e.g., schema >>> >>> versions, >>> >>> > > > > > >> state >>> >>> > > > > > >>>>> entry >>> >>> > > > > > >>>>>>>>>> counts) >>> >>> > > > > > >>>>>>>>>>>> across catalogs reveals structural or >>> quantitative >>> >>> > > > > > >>>>> differences. >>> >>> > > > > > >>>>>>>>>>>> 3. For deeper analysis, users could write SQL >>> >>> queries >>> >>> > to >>> >>> > > > > > >>>>> compare >>> >>> > > > > > >>>>>>>>>> specific >>> >>> > > > > > >>>>>>>>>>>> state partitions or leverage the metaservice >>> to >>> >>> track >>> >>> > > > > > >> state >>> >>> > > > > > >>>>>>>> evolution >>> >>> > > > > > >>>>>>>>>>>> (e.g., added/removed operators, modified state >>> >>> > > > > > >>>>> configurations). >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> If we plan to introduce a state catalog in the >>> >>> > future, I >>> >>> > > > > > >>>> would >>> >>> > > > > > >>>>>>> lean >>> >>> > > > > > >>>>>>>>>>> toward >>> >>> > > > > > >>>>>>>>>>>> using metadata tables. If a utility tool can >>> >>> address >>> >>> > the >>> >>> > > > > > >>>>>>> challenges >>> >>> > > > > > >>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>> face, could we avoid introducing an additional >>> >>> > > connector? >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> >>> 于2025年3月17日周一 >>> >>> > > 20:25写道： >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> Hi All! >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> Without going into too much detail here are >>> my 2 >>> >>> > cents >>> >>> > > > > > >>>>>>> regarding >>> >>> > > > > > >>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> virtual column / catalog metadata / table >>> >>> (connector) >>> >>> > > > > > >>>>>>> discussion >>> >>> > > > > > >>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> State metadata. >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> State metadata such as the types of states, >>> their >>> >>> > > > > > >>>>> properties, >>> >>> > > > > > >>>>>>>>> names, >>> >>> > > > > > >>>>>>>>>>>> sizes >>> >>> > > > > > >>>>>>>>>>>>> etc are all valuable information that can be >>> >>> used to >>> >>> > > > > > >>>> enrich >>> >>> > > > > > >>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> computations we do on state. >>> >>> > > > > > >>>>>>>>>>>>> We can either analyze it standalone (such as >>> >>> discover >>> >>> > > > > > >>>>>>> anomalies, >>> >>> > > > > > >>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>> large >>> >>> > > > > > >>>>>>>>>>>>> jobs with many states), across multiple >>> >>> savepoints >>> >>> > > > > > >>>> (discover >>> >>> > > > > > >>>>>>> how >>> >>> > > > > > >>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>> changed over time) or by joining it with >>> keyed or >>> >>> > > > > > >>>> non-keyed >>> >>> > > > > > >>>>>>> state >>> >>> > > > > > >>>>>>>>>> data >>> >>> > > > > > >>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>> serve more complex queries on the state. >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> The only solution that seems to serve all >>> these >>> >>> > > > > > >> use-cases >>> >>> > > > > > >>>>> and >>> >>> > > > > > >>>>>>>>>>>> requirements >>> >>> > > > > > >>>>>>>>>>>>> in a straightforward and SQL canonical way >>> is to >>> >>> > simply >>> >>> > > > > > >>>>> expose >>> >>> > > > > > >>>>>>>> the >>> >>> > > > > > >>>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>> metadata as a separate table. This is a >>> metadata >>> >>> > table >>> >>> > > > > > >>>> but >>> >>> > > > > > >>>>> you >>> >>> > > > > > >>>>>>>> can >>> >>> > > > > > >>>>>>>>>> also >>> >>> > > > > > >>>>>>>>>>>>> think of it as data table, it makes no >>> practical >>> >>> > > > > > >>>> difference >>> >>> > > > > > >>>>>>> here. >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> Once we have a catalog later, the catalog can >>> >>> offer >>> >>> > > > > > >> this >>> >>> > > > > > >>>>> table >>> >>> > > > > > >>>>>>>> out >>> >>> > > > > > >>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> box, the same way databases provide metadata >>> >>> tables. >>> >>> > > > > > >> For >>> >>> > > > > > >>>>> this >>> >>> > > > > > >>>>>>> to >>> >>> > > > > > >>>>>>>>> work >>> >>> > > > > > >>>>>>>>>>>>> however we need another, simpler connector >>> that >>> >>> > creates >>> >>> > > > > > >>>> this >>> >>> > > > > > >>>>>>>> table. >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> +1 for state metadata as a separate >>> >>> connector/table, >>> >>> > > > > > >>>> instead >>> >>> > > > > > >>>>>>> of >>> >>> > > > > > >>>>>>>>>> adding >>> >>> > > > > > >>>>>>>>>>>>> virtual columns and adhoc catalog metadata >>> that >>> >>> is >>> >>> > hard >>> >>> > > > > > >>>> to >>> >>> > > > > > >>>>> use >>> >>> > > > > > >>>>>>>> in a >>> >>> > > > > > >>>>>>>>>>> large >>> >>> > > > > > >>>>>>>>>>>>> number of queries. >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> Cheers, >>> >>> > > > > > >>>>>>>>>>>>> Gyula >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> On Mon, Mar 17, 2025 at 12:44 PM Gabor >>> Somogyi < >>> >>> > > > > > >>>>>>>>>>>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> I’m planning on adding this, and we may >>> >>> collaborate >>> >>> > > > > > >>>> on >>> >>> > > > > > >>>>> it >>> >>> > > > > > >>>>>>> in >>> >>> > > > > > >>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> future. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> +1 on this, just ping me. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> After some code digging and POC all I can >>> say >>> >>> that >>> >>> > > > > > >> with >>> >>> > > > > > >>>>>>> heavy >>> >>> > > > > > >>>>>>>>>> effort >>> >>> > > > > > >>>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>> maybe add such changes that we're able to >>> show >>> >>> > > > > > >> metadata >>> >>> > > > > > >>>>> of a >>> >>> > > > > > >>>>>>>>>>> savepoint >>> >>> > > > > > >>>>>>>>>>>>> from >>> >>> > > > > > >>>>>>>>>>>>>> catalog. >>> >>> > > > > > >>>>>>>>>>>>>> I'm not against that but from user >>> perspective >>> >>> this >>> >>> > > > > > >> has >>> >>> > > > > > >>>>>>> limited >>> >>> > > > > > >>>>>>>>>>> value, >>> >>> > > > > > >>>>>>>>>>>>> let >>> >>> > > > > > >>>>>>>>>>>>>> me explain why. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> From high level perspective I see the >>> following >>> >>> > > > > > >> which I >>> >>> > > > > > >>>>> see >>> >>> > > > > > >>>>>>>>>> agreement >>> >>> > > > > > >>>>>>>>>>>> on: >>> >>> > > > > > >>>>>>>>>>>>>> * We should have a catalog which is >>> >>> representing one >>> >>> > > > > > >> or >>> >>> > > > > > >>>>> more >>> >>> > > > > > >>>>>>>> jobs >>> >>> > > > > > >>>>>>>>>>>>> savepoint >>> >>> > > > > > >>>>>>>>>>>>>> data set (future plan) >>> >>> > > > > > >>>>>>>>>>>>>> * Savepoints should be able to be >>> registered in >>> >>> the >>> >>> > > > > > >>>>> catalog >>> >>> > > > > > >>>>>>>> which >>> >>> > > > > > >>>>>>>>>> are >>> >>> > > > > > >>>>>>>>>>>>> then >>> >>> > > > > > >>>>>>>>>>>>>> databases (future plan) >>> >>> > > > > > >>>>>>>>>>>>>> * There must be a possiblity to create >>> tables >>> >>> from >>> >>> > > > > > >>>>> databases >>> >>> > > > > > >>>>>>>>> where >>> >>> > > > > > >>>>>>>>>>>> users >>> >>> > > > > > >>>>>>>>>>>>>> can read state data (exists already) >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> In terms of metadata, If I understand >>> correctly >>> >>> then >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>> suggested >>> >>> > > > > > >>>>>>>>>>>>> approach >>> >>> > > > > > >>>>>>>>>>>>>> would be to access >>> >>> > > > > > >>>>>>>>>>>>>> it from the catalog describe command, right? >>> >>> Adding >>> >>> > > > > > >>>> that >>> >>> > > > > > >>>>>>> info >>> >>> > > > > > >>>>>>>>> when >>> >>> > > > > > >>>>>>>>>>>>> specific >>> >>> > > > > > >>>>>>>>>>>>>> database describe command >>> >>> > > > > > >>>>>>>>>>>>>> is executed could be done. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> The question is for instance how can users >>> >>> create >>> >>> > > > > > >> such >>> >>> > > > > > >>>> a >>> >>> > > > > > >>>>>>> logic >>> >>> > > > > > >>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>> tells >>> >>> > > > > > >>>>>>>>>>>>>> them what is >>> >>> > > > > > >>>>>>>>>>>>>> the difference between multiple savepoints? >>> >>> > > > > > >>>>>>>>>>>>>> Just to give some examples: >>> >>> > > > > > >>>>>>>>>>>>>> * per operator size changes between >>> savepoints >>> >>> > > > > > >>>>>>>>>>>>>> * show values from operator data where state >>> >>> size >>> >>> > > > > > >>>> reaches >>> >>> > > > > > >>>>> a >>> >>> > > > > > >>>>>>>>>> boundary >>> >>> > > > > > >>>>>>>>>>>>>> * in general "find which checkpoint ruined >>> >>> things" >>> >>> > is >>> >>> > > > > > >>>>> quite >>> >>> > > > > > >>>>>>>>> common >>> >>> > > > > > >>>>>>>>>>>>> pattern >>> >>> > > > > > >>>>>>>>>>>>>> What I would like to highlight here is that >>> from >>> >>> > > > > > >> Flink >>> >>> > > > > > >>>>>>> point of >>> >>> > > > > > >>>>>>>>>> view >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> metadata can be >>> >>> > > > > > >>>>>>>>>>>>>> considered as a static side output >>> information >>> >>> but >>> >>> > > > > > >> for >>> >>> > > > > > >>>>> users >>> >>> > > > > > >>>>>>>>> these >>> >>> > > > > > >>>>>>>>>>>> values >>> >>> > > > > > >>>>>>>>>>>>>> are actual real data >>> >>> > > > > > >>>>>>>>>>>>>> where logic is planned to build around. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> The metadata is more like one-time >>> information >>> >>> > > > > > >>>> instead >>> >>> > > > > > >>>>> of >>> >>> > > > > > >>>>>>> a >>> >>> > > > > > >>>>>>>>>>> streaming >>> >>> > > > > > >>>>>>>>>>>>>> data that changes all >>> >>> > > > > > >>>>>>>>>>>>>> the time, so a single connector seems to be >>> an >>> >>> > > > > > >>>> overkill. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> State data is also static within a >>> savepoint and >>> >>> > > > > > >> that's >>> >>> > > > > > >>>>> the >>> >>> > > > > > >>>>>>>>> reason >>> >>> > > > > > >>>>>>>>>>> why >>> >>> > > > > > >>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> state processor API is working in batch >>> mode. >>> >>> > > > > > >>>>>>>>>>>>>> When we handle multiple checkpoints in a >>> >>> streaming >>> >>> > > > > > >>>> fashion >>> >>> > > > > > >>>>>>> then >>> >>> > > > > > >>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>> viewed from another angle. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> We can come up with more lightweight >>> solution >>> >>> other >>> >>> > > > > > >>>> than a >>> >>> > > > > > >>>>>>> new >>> >>> > > > > > >>>>>>>>>>>> connector >>> >>> > > > > > >>>>>>>>>>>>>> but enforcing users to parse the catalog >>> >>> > > > > > >>>>>>>>>>>>>> describe command output in order to compare >>> >>> multiple >>> >>> > > > > > >>>>>>> savepoints >>> >>> > > > > > >>>>>>>>>>> doesn't >>> >>> > > > > > >>>>>>>>>>>>>> sound smooth user experience. >>> >>> > > > > > >>>>>>>>>>>>>> Honestly I've no other idea how exposing >>> >>> metadata as >>> >>> > > > > > >>>> real >>> >>> > > > > > >>>>>>> user >>> >>> > > > > > >>>>>>>>> data >>> >>> > > > > > >>>>>>>>>>> so >>> >>> > > > > > >>>>>>>>>>>>>> waiting on other approaches. >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> On Thu, Mar 13, 2025 at 2:44 AM Shengkai >>> Fang < >>> >>> > > > > > >>>>>>>> fskm...@gmail.com >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> Looking forward to hearing the good news! >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> Gabor Somogyi <gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>> 于2025年3月12日周三 >>> >>> > > > > > >>>>>>>>> 22:24写道： >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> Thanks for both the valuable input! >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> Let me take a closer look at the >>> suggestions, >>> >>> > > > > > >> like >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>> Catalog >>> >>> > > > > > >>>>>>>>>>>>>>> capabilities >>> >>> > > > > > >>>>>>>>>>>>>>>> and possibility of embedding >>> TypeInformation >>> >>> or >>> >>> > > > > > >>>>>>>>>>>>>>>> StateDescriptor metadata directly into >>> the raw >>> >>> > > > > > >>>> state >>> >>> > > > > > >>>>>>>> files... >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 8:17 AM Shengkai >>> Fang >>> >>> < >>> >>> > > > > > >>>>>>>>>> fskm...@gmail.com >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> Thanks for Zakelly's clarification. >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> +1 to delay the discussion about this. >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> I’d like to share my perspective on the >>> State >>> >>> > > > > > >>>>> Catalog >>> >>> > > > > > >>>>>>>>>> proposal. >>> >>> > > > > > >>>>>>>>>>>>> While >>> >>> > > > > > >>>>>>>>>>>>>>>>> introducing this capability is >>> beneficial, >>> >>> > > > > > >> there >>> >>> > > > > > >>>> is >>> >>> > > > > > >>>>> a >>> >>> > > > > > >>>>>>>>>> blocker: >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>> current >>> >>> > > > > > >>>>>>>>>>>>>>>>> StateBackend architecture does not permit >>> >>> > > > > > >>>> operators >>> >>> > > > > > >>>>> to >>> >>> > > > > > >>>>>>>>> encode >>> >>> > > > > > >>>>>>>>>>>>>>>>> TypeInformation into the state—it only >>> >>> > > > > > >> preserves >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>>>> Serializer. >>> >>> > > > > > >>>>>>>>>>>>> This >>> >>> > > > > > >>>>>>>>>>>>>>>>> limitation creates an asymmetry, as >>> operators >>> >>> > > > > > >>>> alone >>> >>> > > > > > >>>>>>>> retain >>> >>> > > > > > >>>>>>>>>>>>> knowledge >>> >>> > > > > > >>>>>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>> data structure’s schema. >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> To address this, I suggest allowing >>> operators >>> >>> > > > > > >> to >>> >>> > > > > > >>>>> embed >>> >>> > > > > > >>>>>>>>>>>>>> TypeInformation >>> >>> > > > > > >>>>>>>>>>>>>>> or >>> >>> > > > > > >>>>>>>>>>>>>>>>> StateDescriptor metadata directly into >>> the >>> >>> raw >>> >>> > > > > > >>>> state >>> >>> > > > > > >>>>>>>> files. >>> >>> > > > > > >>>>>>>>>>> Such >>> >>> > > > > > >>>>>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>>>>>> design >>> >>> > > > > > >>>>>>>>>>>>>>>>> would enable the Catalog to: >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> 1. Parse state files and programmatically >>> >>> > > > > > >> derive >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>> schema >>> >>> > > > > > >>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>> structural >>> >>> > > > > > >>>>>>>>>>>>>>>>> guarantees for each state. >>> >>> > > > > > >>>>>>>>>>>>>>>>> 2. Leverage existing Flink Table >>> utilities, >>> >>> > > > > > >> such >>> >>> > > > > > >>>> as >>> >>> > > > > > >>>>>>>>>>>>>>>>> LegacyTypeInfoDataTypeConverter (in >>> >>> > > > > > >>>>>>>>>>>>>>> org.apache.flink.table.types.utils), >>> >>> > > > > > >>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>> bridge TypeInformation and DataType >>> >>> > > > > > >> conversions. >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> If we can not store the TypeInformation >>> or >>> >>> > > > > > >>>>>>>> StateDescriptor >>> >>> > > > > > >>>>>>>>>> into >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> raw >>> >>> > > > > > >>>>>>>>>>>>>>>>> state files, I am +1 for this FLIP to use >>> >>> > > > > > >>>> metadata >>> >>> > > > > > >>>>>>> column >>> >>> > > > > > >>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>> retrieve >>> >>> > > > > > >>>>>>>>>>>>>>>>> information. >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> Zakelly Lan <zakelly....@gmail.com> >>> >>> > > > > > >>>> 于2025年3月12日周三 >>> >>> > > > > > >>>>>>>> 12:43写道： >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Hi Gabor and Shengkai, >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Thanks for sharing your thoughts! This >>> is a >>> >>> > > > > > >>>> long >>> >>> > > > > > >>>>>>>>> discussion >>> >>> > > > > > >>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>> sorry >>> >>> > > > > > >>>>>>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the late reply (I'm busy catching up >>> with >>> >>> > > > > > >>>> release >>> >>> > > > > > >>>>>>> 2.0 >>> >>> > > > > > >>>>>>>>> these >>> >>> > > > > > >>>>>>>>>>>>> days). >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Let me first clarify your thoughts to >>> ensure >>> >>> > > > > > >> I >>> >>> > > > > > >>>>>>>> understand >>> >>> > > > > > >>>>>>>>>>>>>> correctly. >>> >>> > > > > > >>>>>>>>>>>>>>>>> IIUC, >>> >>> > > > > > >>>>>>>>>>>>>>>>>> there is no persistent configuration for >>> >>> > > > > > >> state >>> >>> > > > > > >>>> TTL >>> >>> > > > > > >>>>>>> in >>> >>> > > > > > >>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> checkpoint. >>> >>> > > > > > >>>>>>>>>>>>>>>>> While >>> >>> > > > > > >>>>>>>>>>>>>>>>>> you can infer that TTL is enabled by >>> reading >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>>> serializer, >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>> checkpoint >>> >>> > > > > > >>>>>>>>>>>>>>>>>> itself only stores the last access time >>> for >>> >>> > > > > > >>>> each >>> >>> > > > > > >>>>>>> value. >>> >>> > > > > > >>>>>>>>> So >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> only >>> >>> > > > > > >>>>>>>>>>>>>>>> thing >>> >>> > > > > > >>>>>>>>>>>>>>>>>> we can show is the last access time for >>> each >>> >>> > > > > > >>>>> value. >>> >>> > > > > > >>>>>>> But >>> >>> > > > > > >>>>>>>>> it >>> >>> > > > > > >>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>>>>>>> required >>> >>> > > > > > >>>>>>>>>>>>>>>>>> for all state backends to store this, as >>> >>> they >>> >>> > > > > > >>>> may >>> >>> > > > > > >>>>>>>>> directly >>> >>> > > > > > >>>>>>>>>>>> store >>> >>> > > > > > >>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>> expired time. This will also increase >>> the >>> >>> > > > > > >>>>>>> difficulty of >>> >>> > > > > > >>>>>>>>>>>>>>> implementation >>> >>> > > > > > >>>>>>>>>>>>>>>> & >>> >>> > > > > > >>>>>>>>>>>>>>>>>> maintenance. >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> This once again reiterates the >>> importance of >>> >>> > > > > > >>>>> unified >>> >>> > > > > > >>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>>>>> checkpoints. I’m planning on adding >>> this, >>> >>> and >>> >>> > > > > > >>>> we >>> >>> > > > > > >>>>> may >>> >>> > > > > > >>>>>>>>>>>> collaborate >>> >>> > > > > > >>>>>>>>>>>>> on >>> >>> > > > > > >>>>>>>>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the future. >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> I'm not in favor of adding a new >>> connector >>> >>> > > > > > >> for >>> >>> > > > > > >>>>>>>> metadata. >>> >>> > > > > > >>>>>>>>>> The >>> >>> > > > > > >>>>>>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>>>>>>>> more like one-time information instead >>> of a >>> >>> > > > > > >>>>>>> streaming >>> >>> > > > > > >>>>>>>>> data >>> >>> > > > > > >>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>> changes >>> >>> > > > > > >>>>>>>>>>>>>>>>> all >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the time, so a single connector seems >>> to be >>> >>> > > > > > >> an >>> >>> > > > > > >>>>>>>> overkill. >>> >>> > > > > > >>>>>>>>> It >>> >>> > > > > > >>>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>>>>> easy >>> >>> > > > > > >>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>> withdraw a connector if we have a better >>> >>> > > > > > >>>> solution >>> >>> > > > > > >>>>> in >>> >>> > > > > > >>>>>>>>>> future. >>> >>> > > > > > >>>>>>>>>>>> I'm >>> >>> > > > > > >>>>>>>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>>>>>>>> familiar with current Catalog >>> capabilities, >>> >>> > > > > > >>>> and if >>> >>> > > > > > >>>>>>> it >>> >>> > > > > > >>>>>>>>> could >>> >>> > > > > > >>>>>>>>>>>>> extract >>> >>> > > > > > >>>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>> show some operator-level information >>> from >>> >>> > > > > > >>>>> savepoint, >>> >>> > > > > > >>>>>>>> that >>> >>> > > > > > >>>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>> great. >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> If the Catalog can't do that, I would >>> >>> > > > > > >> consider >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>> current >>> >>> > > > > > >>>>>>>>>>> FLIP >>> >>> > > > > > >>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>> be a >>> >>> > > > > > >>>>>>>>>>>>>>>>>> compromise solution. >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> And if we have that unified metadata for >>> >>> > > > > > >>>>>>>>>> checkpoint/savepoint >>> >>> > > > > > >>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>> future, >>> >>> > > > > > >>>>>>>>>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>>>>>>>> may directly register savepoint in >>> catalog, >>> >>> > > > > > >> and >>> >>> > > > > > >>>>>>> create >>> >>> > > > > > >>>>>>>> a >>> >>> > > > > > >>>>>>>>>>> source >>> >>> > > > > > >>>>>>>>>>>>>>> without >>> >>> > > > > > >>>>>>>>>>>>>>>>>> specifying complex columns, as well as >>> >>> > > > > > >> describe >>> >>> > > > > > >>>>> the >>> >>> > > > > > >>>>>>>>>> savepoint >>> >>> > > > > > >>>>>>>>>>>>>> catalog >>> >>> > > > > > >>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>> get the metadata. That's a good >>> solution in >>> >>> > > > > > >> my >>> >>> > > > > > >>>>> mind. >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Zakelly >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 10:35 AM >>> Shengkai >>> >>> > > > > > >> Fang >>> >>> > > > > > >>>> < >>> >>> > > > > > >>>>>>>>>>>>> fskm...@gmail.com> >>> >>> > > > > > >>>>>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Hi Gabor, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>> >>> > > > > > >>>>>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> I would argue against introducing a new >>> >>> > > > > > >>>>> connector >>> >>> > > > > > >>>>>>>> type >>> >>> > > > > > >>>>>>>>>>> named >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> savepoint-metadata, as the existing >>> Catalog >>> >>> > > > > > >>>>>>> mechanism >>> >>> > > > > > >>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>> inherently >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> provide the necessary connector factory >>> >>> > > > > > >>>>>>> capabilities. >>> >>> > > > > > >>>>>>>>>> I’ve >>> >>> > > > > > >>>>>>>>>>>>>> detailed >>> >>> > > > > > >>>>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> proposal in branch[1]. Please take a >>> moment >>> >>> > > > > > >>>> to >>> >>> > > > > > >>>>>>> review >>> >>> > > > > > >>>>>>>>> it. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> If we introduce a connector named >>> >>> > > > > > >>>>>>>> `savepoint-metadata`, >>> >>> > > > > > >>>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>>> means >>> >>> > > > > > >>>>>>>>>>>>>>> user >>> >>> > > > > > >>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> create a temporary table with connector >>> >>> > > > > > >>>>>>>>>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> connector needs to check whether table >>> >>> > > > > > >>>> schema is >>> >>> > > > > > >>>>>>> same >>> >>> > > > > > >>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> schema >>> >>> > > > > > >>>>>>>>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> proposed in the FLIP. On the other >>> hand, >>> >>> > > > > > >> it's >>> >>> > > > > > >>>>> not >>> >>> > > > > > >>>>>>>> easy >>> >>> > > > > > >>>>>>>>>> work >>> >>> > > > > > >>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>>> others >>> >>> > > > > > >>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> users a metadata table with same >>> schema. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://github.com/apache/flink/compare/master...fsk119:flink:state-metadata?expand=1#diff-712a7bc92fe46c405fb0e61b475bb2a005cb7a72bab7df28bbb92744bcb5f465R63 >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Gabor Somogyi < >>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>>>>>> 于2025年3月11日周二 >>> >>> > > > > > >>>>>>>>>>>>> 16:56写道： >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Shengkai, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> From directional perspective I agree >>> your >>> >>> > > > > > >>>> idea >>> >>> > > > > > >>>>>>> how >>> >>> > > > > > >>>>>>>> it >>> >>> > > > > > >>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>>>> implemented. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Previously I've mentioned that TTL >>> >>> > > > > > >>>> information >>> >>> > > > > > >>>>>>> is >>> >>> > > > > > >>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>> exposed >>> >>> > > > > > >>>>>>>>>>>>>> on >>> >>> > > > > > >>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> processor API (which the SQL state >>> >>> > > > > > >>>> connector >>> >>> > > > > > >>>>>>> uses >>> >>> > > > > > >>>>>>>> to >>> >>> > > > > > >>>>>>>>>> read >>> >>> > > > > > >>>>>>>>>>>>> data) >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> and unless somebody show me the >>> opposite >>> >>> > > > > > >>>> this >>> >>> > > > > > >>>>>>> FLIP >>> >>> > > > > > >>>>>>>> is >>> >>> > > > > > >>>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>>> going >>> >>> > > > > > >>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> address >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> this to avoid feature creep. Our users >>> >>> > > > > > >> are >>> >>> > > > > > >>>>> also >>> >>> > > > > > >>>>>>>>>>> interested >>> >>> > > > > > >>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>> TTL >>> >>> > > > > > >>>>>>>>>>>>>>>> so >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> sooner or later we're going to expose >>> it, >>> >>> > > > > > >>>> this >>> >>> > > > > > >>>>>>> is >>> >>> > > > > > >>>>>>>>>> matter >>> >>> > > > > > >>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>>> scheduling. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>> >>> > > > > > >>>>>>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Not sure I understand your point at >>> all >>> >>> > > > > > >>>>> related >>> >>> > > > > > >>>>>>>>>>>> StateCatalog. >>> >>> > > > > > >>>>>>>>>>>>>>> First >>> >>> > > > > > >>>>>>>>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> all >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I can't agree more that StateCatalog >>> is >>> >>> > > > > > >>>> needed >>> >>> > > > > > >>>>>>> and >>> >>> > > > > > >>>>>>>>> is a >>> >>> > > > > > >>>>>>>>>>>>> planned >>> >>> > > > > > >>>>>>>>>>>>>>>>>> building >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> block in an upcoming >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> FLIP but not sure how can it help >>> now? No >>> >>> > > > > > >>>>> matter >>> >>> > > > > > >>>>>>>>> what, >>> >>> > > > > > >>>>>>>>>>> your >>> >>> > > > > > >>>>>>>>>>>>>>>> knowledge >>> >>> > > > > > >>>>>>>>>>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> essential when we add StateCatalog. >>> Let >>> >>> > > > > > >> me >>> >>> > > > > > >>>>>>> expose >>> >>> > > > > > >>>>>>>> my >>> >>> > > > > > >>>>>>>>>>>>>>> understanding >>> >>> > > > > > >>>>>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> area: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * First we need create table >>> statements >>> >>> > > > > > >> to >>> >>> > > > > > >>>>>>> access >>> >>> > > > > > >>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>> data >>> >>> > > > > > >>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * When we have that then we can add >>> >>> > > > > > >>>>> StateCatalog >>> >>> > > > > > >>>>>>>>> which >>> >>> > > > > > >>>>>>>>>>>> could >>> >>> > > > > > >>>>>>>>>>>>>>>>>> potentially >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> ease the life of users by for ex. >>> giving >>> >>> > > > > > >>>>>>>>> off-the-shelf >>> >>> > > > > > >>>>>>>>>>>> tables >>> >>> > > > > > >>>>>>>>>>>>>>>> without >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> sweating with create table statements >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> User expectations: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See state data (this is fulfilled >>> with >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>> existing >>> >>> > > > > > >>>>>>>>>>>>>> connector) >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See metadata about state data like >>> TTL >>> >>> > > > > > >>>> (this >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>> be >>> >>> > > > > > >>>>>>>>>>> added >>> >>> > > > > > >>>>>>>>>>>>> as >>> >>> > > > > > >>>>>>>>>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> column as you suggested since it >>> belongs >>> >>> > > > > > >> to >>> >>> > > > > > >>>>> the >>> >>> > > > > > >>>>>>>> data) >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See metadata about operators (this >>> can >>> >>> > > > > > >> be >>> >>> > > > > > >>>>>>> added >>> >>> > > > > > >>>>>>>>> from >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> savepoint-metadata) >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Important to highlight that state data >>> >>> > > > > > >>>> table >>> >>> > > > > > >>>>>>> format >>> >>> > > > > > >>>>>>>>>>> differs >>> >>> > > > > > >>>>>>>>>>>>>> from >>> >>> > > > > > >>>>>>>>>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> metadata table format. Namely one >>> table >>> >>> > > > > > >> has >>> >>> > > > > > >>>>> rows >>> >>> > > > > > >>>>>>>> for >>> >>> > > > > > >>>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>>> values >>> >>> > > > > > >>>>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> another has rows for operators, right? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I think that's the reason why you've >>> >>> > > > > > >>>>> pinpointed >>> >>> > > > > > >>>>>>> out >>> >>> > > > > > >>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>> suggested >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> metadata columns are somewhat clunky. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> As a conclusion I agree to add >>> >>> > > > > > >>>>> ${state-name}_ttl >>> >>> > > > > > >>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>> column >>> >>> > > > > > >>>>>>>>>>>>>>>>> later >>> >>> > > > > > >>>>>>>>>>>>>>>>>> on >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> since it belongs to the state value >>> and >>> >>> > > > > > >>>>> adding a >>> >>> > > > > > >>>>>>>> new >>> >>> > > > > > >>>>>>>>>>> table >>> >>> > > > > > >>>>>>>>>>>>> type >>> >>> > > > > > >>>>>>>>>>>>>>>> (like >>> >>> > > > > > >>>>>>>>>>>>>>>>>> you >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> suggested similar to PG [1]) >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> for metadata. Please see how Spark >>> does >>> >>> > > > > > >>>> that >>> >>> > > > > > >>>>> too >>> >>> > > > > > >>>>>>>> [2]. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> If you have better approach then >>> please >>> >>> > > > > > >>>>>>> elaborate >>> >>> > > > > > >>>>>>>>> with >>> >>> > > > > > >>>>>>>>>>> more >>> >>> > > > > > >>>>>>>>>>>>>>> details >>> >>> > > > > > >>>>>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> help me to understand your point. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>> >>> > > > > > >>>>> savepoints >>> >>> > > > > > >>>>>>>> that >>> >>> > > > > > >>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>> number >>> >>> > > > > > >>>>>>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>>> keys >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key >>> >>> > > > > > >>>> state >>> >>> > > > > > >>>>>>>> itself. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> But again, this is a good feature >>> as-is >>> >>> > > > > > >>>> and >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>> handled >>> >>> > > > > > >>>>>>>>>>>>>> in a >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> separate >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> jira. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I've just created >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >> https://issues.apache.org/jira/browse/FLINK-37456. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>> >>> >>> https://www.postgresql.org/docs/current/view-pg-tables.html >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [2] >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Mar 11, 2025 at 3:55 AM >>> Shengkai >>> >>> > > > > > >>>> Fang >>> >>> > > > > > >>>>> < >>> >>> > > > > > >>>>>>>>>>>>>> fskm...@gmail.com >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your response. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Thank you for addressing the >>> >>> > > > > > >> limitations >>> >>> > > > > > >>>>> here. >>> >>> > > > > > >>>>>>>>>>> However, I >>> >>> > > > > > >>>>>>>>>>>>>>> believe >>> >>> > > > > > >>>>>>>>>>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> be beneficial to further clarify the >>> >>> > > > > > >> API >>> >>> > > > > > >>>> in >>> >>> > > > > > >>>>>>> this >>> >>> > > > > > >>>>>>>>> FLIP >>> >>> > > > > > >>>>>>>>>>>>>> regarding >>> >>> > > > > > >>>>>>>>>>>>>>>> how >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> users >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> can specify the TTL column. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> One potential approach that comes to >>> >>> > > > > > >>>> mind is >>> >>> > > > > > >>>>>>>> using >>> >>> > > > > > >>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>>>>>> standardized >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> naming >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> convention such as ${state-name}_ttl >>> >>> > > > > > >> for >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>> column >>> >>> > > > > > >>>>>>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> defines >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> the TTL value. In terms of >>> >>> > > > > > >>>> implementation, >>> >>> > > > > > >>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>> listReadableMetadata >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> function could: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. Read the table’s columns and >>> >>> > > > > > >>>>> configuration, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Extract all defined state names, >>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 3. Return a structured list of >>> metadata >>> >>> > > > > > >>>>>>> entries >>> >>> > > > > > >>>>>>>>>>> formatted >>> >>> > > > > > >>>>>>>>>>>>> as >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> ${state-name}_ttl. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> WDYT? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>> >>> > > > > > >>>>>>>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Introducing a new connector type at >>> >>> > > > > > >> this >>> >>> > > > > > >>>>> stage >>> >>> > > > > > >>>>>>>> may >>> >>> > > > > > >>>>>>>>>>>>>>> unnecessarily >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> complicate >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> the system. Given that every table >>> >>> > > > > > >>>> already >>> >>> > > > > > >>>>>>>> belongs >>> >>> > > > > > >>>>>>>>>> to a >>> >>> > > > > > >>>>>>>>>>>>>>> Catalog, >>> >>> > > > > > >>>>>>>>>>>>>>>>>> which >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> designed to provide a Factory for >>> >>> > > > > > >>>> building >>> >>> > > > > > >>>>>>> source >>> >>> > > > > > >>>>>>>>> or >>> >>> > > > > > >>>>>>>>>>> sink >>> >>> > > > > > >>>>>>>>>>>>>>>>>> connectors, I >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> propose integrating a dedicated >>> >>> > > > > > >>>> StateCatalog >>> >>> > > > > > >>>>>>>>> instead. >>> >>> > > > > > >>>>>>>>>>>> This >>> >>> > > > > > >>>>>>>>>>>>>>>> approach >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> allow us to: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. Leverage the Catalog’s existing >>> >>> > > > > > >>>>>>> capabilities >>> >>> > > > > > >>>>>>>> to >>> >>> > > > > > >>>>>>>>>>> manage >>> >>> > > > > > >>>>>>>>>>>>> TTL >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> (e.g., state names and TTL logic) >>> >>> > > > > > >> without >>> >>> > > > > > >>>>>>>>> duplicating >>> >>> > > > > > >>>>>>>>>>>>>>>>> functionality. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Provide a unified interface for >>> >>> > > > > > >>>> connector >>> >>> > > > > > >>>>>>>>>>>> instantiation >>> >>> > > > > > >>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> handling through the Catalog’s >>> Factory >>> >>> > > > > > >>>>>>> pattern. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Would this design decision better >>> align >>> >>> > > > > > >>>> with >>> >>> > > > > > >>>>>>> our >>> >>> > > > > > >>>>>>>>>>>>>> architecture’s >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> extensibility and reduce redundancy? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>> >>> > > > > > >>>>>>> savepoints >>> >>> > > > > > >>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> number >>> >>> > > > > > >>>>>>>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>>>> keys >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per >>> key >>> >>> > > > > > >>>>> state >>> >>> > > > > > >>>>>>>>> itself. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature >>> >>> > > > > > >> as-is >>> >>> > > > > > >>>>> and >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>> handled >>> >>> > > > > > >>>>>>>>>>>>>>> in a >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> separate >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> jira. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> +1 for a separate jira. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < >>> >>> > > > > > >> gabor.g.somo...@gmail.com >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>>>>>>>>> 于2025年3月10日周一 >>> >>> > > > > > >>>>>>>>>>>>>>> 19:05写道： >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Shengkai, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Please see my comments inline. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 3, 2025 at 7:07 AM >>> >>> > > > > > >> Shengkai >>> >>> > > > > > >>>>>>> Fang < >>> >>> > > > > > >>>>>>>>>>>>>>>> fskm...@gmail.com> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your the >>> >>> > > > > > >> FLIP. >>> >>> > > > > > >>>> I >>> >>> > > > > > >>>>>>> have >>> >>> > > > > > >>>>>>>>> some >>> >>> > > > > > >>>>>>>>>>>>>> questions >>> >>> > > > > > >>>>>>>>>>>>>>>>> about >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> FLIP: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> How can users retrieve the state >>> >>> > > > > > >> TTL >>> >>> > > > > > >>>>>>>>>> (Time-to-Live) >>> >>> > > > > > >>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>> each >>> >>> > > > > > >>>>>>>>>>>>>>>>>> value >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> column? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> From my understanding of the >>> >>> > > > > > >> current >>> >>> > > > > > >>>>>>> design, >>> >>> > > > > > >>>>>>>> it >>> >>> > > > > > >>>>>>>>>>> seems >>> >>> > > > > > >>>>>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> functionality is not supported. >>> >>> > > > > > >> Could >>> >>> > > > > > >>>>> you >>> >>> > > > > > >>>>>>>>> clarify >>> >>> > > > > > >>>>>>>>>>> if >>> >>> > > > > > >>>>>>>>>>>>>> there >>> >>> > > > > > >>>>>>>>>>>>>>>> are >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> plans >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> address this limitation? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Since the state processor API is not >>> >>> > > > > > >>>> yet >>> >>> > > > > > >>>>>>>> exposing >>> >>> > > > > > >>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>> information >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> would require several steps. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> First, the state processor API >>> >>> > > > > > >> support >>> >>> > > > > > >>>>>>> needs to >>> >>> > > > > > >>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>> added >>> >>> > > > > > >>>>>>>>>>>>>>> which >>> >>> > > > > > >>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> then >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> exposed on the SQL API. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This is definitely a future >>> >>> > > > > > >> improvement >>> >>> > > > > > >>>>>>> which >>> >>> > > > > > >>>>>>>> is >>> >>> > > > > > >>>>>>>>>>> useful >>> >>> > > > > > >>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> handled >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> in a separate jira. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata >>> >>> > > > > > >> Column >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> The metadata information described >>> >>> > > > > > >> in >>> >>> > > > > > >>>>> the >>> >>> > > > > > >>>>>>>> FLIP >>> >>> > > > > > >>>>>>>>>>>> appears >>> >>> > > > > > >>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> intended >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> describe the state files stored at >>> >>> > > > > > >> a >>> >>> > > > > > >>>>>>> specific >>> >>> > > > > > >>>>>>>>>>>> location. >>> >>> > > > > > >>>>>>>>>>>>>> To >>> >>> > > > > > >>>>>>>>>>>>>>>> me, >>> >>> > > > > > >>>>>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> concept >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> aligns more closely with system >>> >>> > > > > > >>>> tables >>> >>> > > > > > >>>>>>> like >>> >>> > > > > > >>>>>>>>>>> pg_tables >>> >>> > > > > > >>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>> PostgreSQL >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> or >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the INFORMATION_SCHEMA in MySQL >>> >>> > > > > > >> [2]. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Adding a new connector with >>> >>> > > > > > >>>>>>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>> is a >>> >>> > > > > > >>>>>>>>>>>>>>>> possibility >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> where >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> can create such functionality. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> I'm not against that, just want to >>> >>> > > > > > >>>> have a >>> >>> > > > > > >>>>>>>> common >>> >>> > > > > > >>>>>>>>>>>>> agreement >>> >>> > > > > > >>>>>>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>>>> we >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> like to move that direction. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> (As a side note not just PG but >>> Spark >>> >>> > > > > > >>>> also >>> >>> > > > > > >>>>>>> has >>> >>> > > > > > >>>>>>>>>>> similar >>> >>> > > > > > >>>>>>>>>>>>>>> approach >>> >>> > > > > > >>>>>>>>>>>>>>>>>> and I >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> basically like the idea). >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> If we would go that direction >>> >>> > > > > > >> savepoint >>> >>> > > > > > >>>>>>>> metadata >>> >>> > > > > > >>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>> reached >>> >>> > > > > > >>>>>>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>> a >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> way >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> that one row would represent >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> an operator with it's values >>> >>> > > > > > >> something >>> >>> > > > > > >>>>> like >>> >>> > > > > > >>>>>>>> this: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┐ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> │operatorN│operatorU│operatorH│paralleli│maxParall│subtaskSt│coordinat│totalSta│ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ame │id │ash │sm >>> >>> > > > > > >>>>>>> │elism >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │atesCount│orStateSi│tesSizeI│ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │ │ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │zeInBytes│nBytes │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │Source: │datagen-s│47aee9439│2 >>> >>> > > > > > >>>>> │128 >>> >>> > > > > > >>>>>>>>>> │2 >>> >>> > > > > > >>>>>>>>>>>>>>> │16 >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │546 │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │datagen-s│ource-uid│4d6ea26e2│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ource │ │d544bef0a│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │37bb5 │ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │long-udf-│long-udf-│6ed3f40bf│2 >>> >>> > > > > > >>>>> │128 >>> >>> > > > > > >>>>>>>>>> │2 >>> >>> > > > > > >>>>>>>>>>>>>>> │0 >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> │0 >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │with-mast│with-mast│f3c8dfcdf│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │er-hook │er-hook-u│cb95128a1│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │id │018f1 │ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │value-pro│value-pro│ca4f5fe9a│2 >>> >>> > > > > > >>>>> │128 >>> >>> > > > > > >>>>>>>>>> │2 >>> >>> > > > > > >>>>>>>>>>>>>>> │0 >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │40726 │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │cess │cess-uid │637b656f0│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │9ea78b3e7│ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │a15b9 │ >>> >>> > > > > > >>>> │ >>> >>> > > > > > >>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This table can then be joined with >>> >>> > > > > > >> the >>> >>> > > > > > >>>>>>> actually >>> >>> > > > > > >>>>>>>>>>>> existing >>> >>> > > > > > >>>>>>>>>>>>>>>>>> `savepoint` >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> connector created tables based on >>> UID >>> >>> > > > > > >>>> hash >>> >>> > > > > > >>>>>>>> (which >>> >>> > > > > > >>>>>>>>>> is >>> >>> > > > > > >>>>>>>>>>>>> unique >>> >>> > > > > > >>>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> always >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> exists). >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This would mean that the already >>> >>> > > > > > >>>> existing >>> >>> > > > > > >>>>>>> table >>> >>> > > > > > >>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>> need >>> >>> > > > > > >>>>>>>>>>>>>>>> only a >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> single >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> metadata column which is the UID >>> >>> > > > > > >> hash. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> WDYT? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> @zakelly, plz share your thoughts >>> >>> > > > > > >> too. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> If we opt to use metadata columns, >>> >>> > > > > > >>>> every >>> >>> > > > > > >>>>>>>> record >>> >>> > > > > > >>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>> table >>> >>> > > > > > >>>>>>>>>>>>>>>>>> would >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> end >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> up >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> having identical values for these >>> >>> > > > > > >>>>> columns >>> >>> > > > > > >>>>>>>>> (please >>> >>> > > > > > >>>>>>>>>>>>> correct >>> >>> > > > > > >>>>>>>>>>>>>>> me >>> >>> > > > > > >>>>>>>>>>>>>>>> if >>> >>> > > > > > >>>>>>>>>>>>>>>>>> I’m >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> mistaken). On the other hand, the >>> >>> > > > > > >>>> state >>> >>> > > > > > >>>>>>>>> connector >>> >>> > > > > > >>>>>>>>>>>>>> requires >>> >>> > > > > > >>>>>>>>>>>>>>>>> users >>> >>> > > > > > >>>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> specify >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> an operator UID or operator UID >>> >>> > > > > > >> hash, >>> >>> > > > > > >>>>>>> after >>> >>> > > > > > >>>>>>>>> which >>> >>> > > > > > >>>>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>>>>> outputs >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> user-defined >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> values in its records. This >>> >>> > > > > > >> approach >>> >>> > > > > > >>>>> feels >>> >>> > > > > > >>>>>>>>>> somewhat >>> >>> > > > > > >>>>>>>>>>>>>>> redundant >>> >>> > > > > > >>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> me. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> If we would add a new >>> >>> > > > > > >>>> `savepoint-metadata` >>> >>> > > > > > >>>>>>>>>> connector >>> >>> > > > > > >>>>>>>>>>>> then >>> >>> > > > > > >>>>>>>>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> addressed. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> On the other hand UID and UID hash >>> >>> > > > > > >> are >>> >>> > > > > > >>>>>>> having >>> >>> > > > > > >>>>>>>>>>> either-or >>> >>> > > > > > >>>>>>>>>>>>>>>>>> relationship >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> from >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> config perspective, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> so when a user provides the UID then >>> >>> > > > > > >>>>> he/she >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>>> interested >>> >>> > > > > > >>>>>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> hash >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> for further calculations >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> (the whole Flink internals are >>> >>> > > > > > >>>> depending >>> >>> > > > > > >>>>> on >>> >>> > > > > > >>>>>>> the >>> >>> > > > > > >>>>>>>>>>> hash). >>> >>> > > > > > >>>>>>>>>>>>>>> Printing >>> >>> > > > > > >>>>>>>>>>>>>>>>> out >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> human readable UID >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> is an explicit requirement from the >>> >>> > > > > > >>>> user >>> >>> > > > > > >>>>>>> side >>> >>> > > > > > >>>>>>>>>> because >>> >>> > > > > > >>>>>>>>>>>>>> hashes >>> >>> > > > > > >>>>>>>>>>>>>>>> are >>> >>> > > > > > >>>>>>>>>>>>>>>>>> not >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> human >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> readable. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 3. Handling LIST and MAP States in >>> >>> > > > > > >>>> the >>> >>> > > > > > >>>>>>> State >>> >>> > > > > > >>>>>>>>>>>> Connector >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I have concerns about how the >>> >>> > > > > > >> current >>> >>> > > > > > >>>>>>> design >>> >>> > > > > > >>>>>>>>>>> handles >>> >>> > > > > > >>>>>>>>>>>>> LIST >>> >>> > > > > > >>>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>> MAP >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> states. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Specifically, the state connector >>> >>> > > > > > >>>> uses >>> >>> > > > > > >>>>>>> Flink >>> >>> > > > > > >>>>>>>>>> SQL’s >>> >>> > > > > > >>>>>>>>>>>> MAP >>> >>> > > > > > >>>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>> ARRAY >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> types, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> which implies that it attempts to >>> >>> > > > > > >>>> load >>> >>> > > > > > >>>>>>> entire >>> >>> > > > > > >>>>>>>>> MAP >>> >>> > > > > > >>>>>>>>>>> or >>> >>> > > > > > >>>>>>>>>>>>> LIST >>> >>> > > > > > >>>>>>>>>>>>>>>>> states >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> into >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> memory. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> However, in many real-world >>> >>> > > > > > >>>> scenarios, >>> >>> > > > > > >>>>>>> these >>> >>> > > > > > >>>>>>>>>> states >>> >>> > > > > > >>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>> grow >>> >>> > > > > > >>>>>>>>>>>>>>>>> very >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> large. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Typically, the state API addresses >>> >>> > > > > > >>>> this >>> >>> > > > > > >>>>> by >>> >>> > > > > > >>>>>>>>>>> providing >>> >>> > > > > > >>>>>>>>>>>> an >>> >>> > > > > > >>>>>>>>>>>>>>>>> iterator >>> >>> > > > > > >>>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> traverse elements within the state >>> >>> > > > > > >>>>>>>>> incrementally. >>> >>> > > > > > >>>>>>>>>>> I’m >>> >>> > > > > > >>>>>>>>>>>>>>> unsure >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> whether >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> I’ve >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> missed something in FLIP-496 or >>> >>> > > > > > >>>>> FLIP-512, >>> >>> > > > > > >>>>>>> but >>> >>> > > > > > >>>>>>>>> it >>> >>> > > > > > >>>>>>>>>>>> seems >>> >>> > > > > > >>>>>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> current >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> design might struggle with >>> >>> > > > > > >>>> scalability >>> >>> > > > > > >>>>> in >>> >>> > > > > > >>>>>>>> such >>> >>> > > > > > >>>>>>>>>>> cases. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> You see it good, the current >>> >>> > > > > > >>>>> implementation >>> >>> > > > > > >>>>>>>> keeps >>> >>> > > > > > >>>>>>>>>>> state >>> >>> > > > > > >>>>>>>>>>>>>> for a >>> >>> > > > > > >>>>>>>>>>>>>>>>>> single >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> key >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> in >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> memory. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Back in the days we've considered >>> >>> > > > > > >> this >>> >>> > > > > > >>>>>>>> potential >>> >>> > > > > > >>>>>>>>>>> issue >>> >>> > > > > > >>>>>>>>>>>>> and >>> >>> > > > > > >>>>>>>>>>>>>>>>>> concluded >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> this is not necessarily >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> needed for the initial version and >>> >>> > > > > > >> can >>> >>> > > > > > >>>> be >>> >>> > > > > > >>>>>>> done >>> >>> > > > > > >>>>>>>>> as a >>> >>> > > > > > >>>>>>>>>>>> later >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> improvement. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>> >>> > > > > > >>>>>>> savepoints >>> >>> > > > > > >>>>>>>>> that >>> >>> > > > > > >>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>> number >>> >>> > > > > > >>>>>>>>>>>>>>>> of >>> >>> > > > > > >>>>>>>>>>>>>>>>>> keys >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> can >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per >>> key >>> >>> > > > > > >>>>> state >>> >>> > > > > > >>>>>>>>> itself. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature >>> >>> > > > > > >> as-is >>> >>> > > > > > >>>>> and >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>>> be >>> >>> > > > > > >>>>>>>>>>>>> handled >>> >>> > > > > > >>>>>>>>>>>>>>> in a >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> separate >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> jira. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Shengkai >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > https://www.postgresql.org/docs/current/view-pg-tables.html >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> [2] >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < >>> >>> > > > > > >>>>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>>>>>>>>> 于2025年3月3日周一 >>> >>> > > > > > >>>>>>>>>>>>>>>>> 02:00写道： >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Zakelly, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> In order to shoot for simplicity >>> >>> > > > > > >>>>>>> `METADATA >>> >>> > > > > > >>>>>>>>>>> VIRTUAL` >>> >>> > > > > > >>>>>>>>>>>>> as >>> >>> > > > > > >>>>>>>>>>>>>>> key >>> >>> > > > > > >>>>>>>>>>>>>>>>>> words >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> definition is the target. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> When it's not super complex the >>> >>> > > > > > >>>> latter >>> >>> > > > > > >>>>>>> can >>> >>> > > > > > >>>>>>>> be >>> >>> > > > > > >>>>>>>>>>> added >>> >>> > > > > > >>>>>>>>>>>>>> too. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 2, 2025 at 3:37 PM >>> >>> > > > > > >>>> Zakelly >>> >>> > > > > > >>>>>>> Lan >>> >>> > > > > > >>>>>>>> < >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> zakelly....@gmail.com> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Gabor, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> +1 for this. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Will the metadata column use >>> >>> > > > > > >>>>> `METADATA >>> >>> > > > > > >>>>>>>>>> VIRTUAL` >>> >>> > > > > > >>>>>>>>>>>> as >>> >>> > > > > > >>>>>>>>>>>>>> key >>> >>> > > > > > >>>>>>>>>>>>>>>>> words >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> definition, or `METADATA FROM >>> >>> > > > > > >> xxx >>> >>> > > > > > >>>>>>>> VIRTUAL` >>> >>> > > > > > >>>>>>>>>> for >>> >>> > > > > > >>>>>>>>>>>>>>> renaming, >>> >>> > > > > > >>>>>>>>>>>>>>>>> just >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> like >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> the >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Kafka table? >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Zakelly >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Mar 1, 2025 at 1:31 PM >>> >>> > > > > > >>>> Gabor >>> >>> > > > > > >>>>>>>>> Somogyi >>> >>> > > > > > >>>>>>>>>> < >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> gabor.g.somo...@gmail.com> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start a >>> >>> > > > > > >> discussion >>> >>> > > > > > >>>> of >>> >>> > > > > > >>>>>>>>> FLIP-512: >>> >>> > > > > > >>>>>>>>>>> Add >>> >>> > > > > > >>>>>>>>>>>>>> meta >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> information >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> to >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> SQL >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> state connector [1]. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to add your >>> >>> > > > > > >> thoughts >>> >>> > > > > > >>>> to >>> >>> > > > > > >>>>>>> make >>> >>> > > > > > >>>>>>>>> this >>> >>> > > > > > >>>>>>>>>>>>> feature >>> >>> > > > > > >>>>>>>>>>>>>>>>> better. >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> G >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>>> >>> >>> > > > > > >>>>>>>>>> >>> >>> > > > > > >>>>>>>>> >>> >>> > > > > > >>>>>>>> >>> >>> > > > > > >>>>>>> >>> >>> > > > > > >>>>>> >>> >>> > > > > > >>>>> >>> >>> > > > > > >>>> >>> >>> > > > > > >>> >>> >>> > > > > > >> >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >>> > > >>> >>> > >>> >>> >>> >> >>> >>
