Yeah, definitely since this a forming feature. BR, G
On Mon, Mar 31, 2025 at 9:41 AM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Gabor, > > Only one minor suggestion, is it possible to make this \@Experimental? I > could expect some new metadata columns added or changed in the near future, > just want to maintain some flexibility until the feature reaches maturity. > > > Best, > Zakelly > > On Fri, Mar 28, 2025 at 8:47 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Just to avoid jumping from one thread to another here's Timo's > suggestion: > > > > "if I understand the discussion correctly, you want to use a PTF without > > table arguments to return a table (read from savepoint metadata)? If > > this is the case, you don't need a PTF for it. A regular table function > > can also do the job. IIRC we support TVF with constant args." > > > > I've tried the TVF out and works with batch mode like charm. > > > > > I'd also suggest we make it built-in without registration. > > I basically agree to have this as built-in. > > > > From my perspective there are no further questions/concerns and updated > the > > FLIP accordingly. > > If somebody has then please share, otherwise I would like to go on with > the > > vote. > > > > All in all thanks for everybody for the constructive suggestions, we've > > made things really better. > > > > BR, > > G > > > > > > On Fri, Mar 28, 2025 at 9:19 AM Zakelly Lan <zakelly....@gmail.com> > wrote: > > > > > Hi all, > > > > > > Given the simplicity, I also +1 for PTF or any other function > > > implementation if PTF is not applicable for this. > > > > > > I would like to raise a consideration regarding the usage > implementation: > > > > Would it be necessary to allow users to utilize the CREATE FUNCTION > > > > statement for registering the PTF? > > > > > > > > > I'd also suggest we make it built-in without registration. > > > > > > Currently, Flink SQL supports letting external systems register modules > > and > > > > leverage these modules to centrally manage all function definitions. > > > Given > > > > this architectural approach, I’m curious if the plan involves > > introducing > > > > additional functions in the future. If so, I would advocate for > > > introducing > > > > a dedicated state module to centralize such management. This would > > > empower > > > > users to: > > > > > > > > > I can’t think of any further functions for now, but I'd +1 for a module > > if > > > it could omit the registration. > > > > > > > > > Best, > > > Zakelly. > > > > > > > > > > > > On Fri, Mar 28, 2025 at 10:25 AM Shengkai Fang <fskm...@gmail.com> > > wrote: > > > > > > > One more question about the FLIP. > > > > > > > > I think the output schema is definitely a public API to users. If > users > > > > use the `CREATE FUNCTION` statement, is it means the class path is > > also a > > > > public API to users. Alternatively, this is merely an experimental > > > feature > > > > and we don't have any promise about this function. > > > > > > > > Best, > > > > Shengkai > > > > > > > > Shengkai Fang <fskm...@gmail.com> 于2025年3月28日周五 10:20写道： > > > > > > > >> +1 to use PTF. > > > >> > > > >> I would like to raise a consideration regarding the usage > > > implementation: > > > >> Would it be necessary to allow users to utilize the CREATE FUNCTION > > > >> statement for registering the PTF? > > > >> > > > >> Currently, Flink SQL supports letting external systems register > > modules > > > >> and leverage these modules to centrally manage all function > > definitions. > > > >> Given this architectural approach, I’m curious if the plan involves > > > >> introducing additional functions in the future. If so, I would > > advocate > > > for > > > >> introducing a dedicated state module to centralize such management. > > This > > > >> would empower users to: > > > >> > > > >> 1. Simply execute the LOAD MODULE command to load the required > module, > > > and > > > >> 2. Directly invoke read_metadata thereafter. > > > >> > > > >> For more details about the module, please refer to this document[1]. > > > >> > > > >> Best, > > > >> Shengkai > > > >> > > > >> [1] > > > >> > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/modules/ > > > >> > > > >> Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月28日周五 00:26写道： > > > >> > > > >>> Just found out that PTF in batch mode is not supported, plz see the > > dev > > > >>> mailing about it [1]. > > > >>> > > > >>> [1] > https://lists.apache.org/thread/ytm9m1qt4pq2q2gjngfktrn8vrlvkf07 > > > >>> > > > >>> BR, > > > >>> G > > > >>> > > > >>> > > > >>> On Thu, Mar 27, 2025 at 3:38 PM Gabor Somogyi < > > > gabor.g.somo...@gmail.com > > > >>> > > > > >>> wrote: > > > >>> > > > >>> > In the meantime I've just updated the FLIP according to this to > be > > > >>> > optimistic 🙂 > > > >>> > > > > >>> > BR, > > > >>> > G > > > >>> > > > > >>> > On Thu, Mar 27, 2025 at 2:15 PM Gabor Somogyi < > > > >>> gabor.g.somo...@gmail.com> > > > >>> > wrote: > > > >>> > > > > >>> >> Considering all the facts I also +1 on PTF. Even if something is > > > >>> missing > > > >>> >> we can add later. > > > >>> >> > > > >>> >> @Zakelly Lan <zakelly....@gmail.com> @Shengkai Fang are you > also > > on > > > >>> the > > > >>> >> same page or have something to add? > > > >>> >> > > > >>> >> BR, > > > >>> >> G > > > >>> >> > > > >>> >> > > > >>> >> On Thu, Mar 27, 2025 at 1:50 PM Lincoln Lee < > > lincoln.8...@gmail.com > > > > > > > >>> >> wrote: > > > >>> >> > > > >>> >>> +1 for PTF > > > >>> >>> > > > >>> >>> > Is it possible to describe such function to see the column > > > >>> names/types? > > > >>> >>> > > > >>> >>> Although Flink SQL does not directly support this feature, > users > > > can > > > >>> >>> achieve > > > >>> >>> similar results with the help of `explain` syntax, e.g. > > > >>> >>> 'explain select * from read_state_metadata(...)' > > > >>> >>> > > > >>> >>> > > > >>> >>> Best, > > > >>> >>> Lincoln Lee > > > >>> >>> > > > >>> >>> > > > >>> >>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月27日周四 20:41写道： > > > >>> >>> > > > >>> >>> > Hey! > > > >>> >>> > > > > >>> >>> > I think the PTF approach strikes a great balance in > simplicity > > > and > > > >>> the > > > >>> >>> > capabilities that we get out of it. > > > >>> >>> > > > > >>> >>> > I think this could be a completely viable alternative to the > > > >>> dedicated > > > >>> >>> > connector, +1. > > > >>> >>> > > > > >>> >>> > Cheers, > > > >>> >>> > Gyula > > > >>> >>> > > > > >>> >>> > On Thu, Mar 27, 2025 at 10:37 AM Shengkai Fang < > > > fskm...@gmail.com> > > > >>> >>> wrote: > > > >>> >>> > > > > >>> >>> > > Hi, Gabor. > > > >>> >>> > > > > > >>> >>> > > > Do I understand correctly that this is 2.x only feature > and > > > we > > > >>> >>> can't > > > >>> >>> > > backport it to 1.x line > > > >>> >>> > > > > > >>> >>> > > Yes. PTF is only supported in 2.x verison. > > > >>> >>> > > > > > >>> >>> > > > Is it possible to describe such function to see the > column > > > >>> >>> names/types? > > > >>> >>> > > > > > >>> >>> > > Flink SQL doesn't support this feature, but postgres[2] or > > > >>> mysql[1] > > > >>> >>> has > > > >>> >>> > > similar feature. > > > >>> >>> > > > > > >>> >>> > > [1] > > > >>> >>> > > https://dev.mysql.com/doc/refman/8.4/en/show-create-procedure.html > > > >>> >>> > > [2] > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://stackoverflow.com/questions/6898453/show-the-code-of-a-function-procedure-and-trigger-in-postgresql > > > >>> >>> > > > > > >>> >>> > > Best, > > > >>> >>> > > Shengkai > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月27日周四 > > > 16:25写道： > > > >>> >>> > > > > > >>> >>> > > > Hi Shengkai, > > > >>> >>> > > > > > > >>> >>> > > > Thanks for your effort with the example, this looks > > > promising. > > > >>> >>> > > > I like the fact that users wouldn't need to sweat with > > > complex > > > >>> >>> create > > > >>> >>> > > table > > > >>> >>> > > > statements. > > > >>> >>> > > > > > > >>> >>> > > > Couple of questions: > > > >>> >>> > > > * Do I understand correctly that this is 2.x only feature > > and > > > >>> we > > > >>> >>> can't > > > >>> >>> > > > backport it to 1.x line? > > > >>> >>> > > > I'm not intended to do any backport, just would like to > > know > > > >>> the > > > >>> >>> > > technical > > > >>> >>> > > > constraints. > > > >>> >>> > > > * Is it possible to describe such function to see the > > column > > > >>> >>> > names/types? > > > >>> >>> > > > > > > >>> >>> > > > BR, > > > >>> >>> > > > G > > > >>> >>> > > > > > > >>> >>> > > > > > > >>> >>> > > > On Thu, Mar 27, 2025 at 3:17 AM Shengkai Fang < > > > >>> fskm...@gmail.com> > > > >>> >>> > wrote: > > > >>> >>> > > > > > > >>> >>> > > > > Many thanks for your reminder, Leonard. Here's the > link I > > > >>> >>> > mentioned[1]. > > > >>> >>> > > > > > > > >>> >>> > > > > Best, > > > >>> >>> > > > > Shengkai > > > >>> >>> > > > > > > > >>> >>> > > > > [1] https://github.com/apache/flink/pull/26358 > > > >>> >>> > > > > > > > >>> >>> > > > > Leonard Xu <xbjt...@gmail.com> 于2025年3月27日周四 10:05写道： > > > >>> >>> > > > > > > > >>> >>> > > > > > Your link is broken, Shengkai > > > >>> >>> > > > > > > > > >>> >>> > > > > > Best, > > > >>> >>> > > > > > Leonard > > > >>> >>> > > > > > > > > >>> >>> > > > > > > 2025年3月27日 10:01，Shengkai Fang <fskm...@gmail.com> > > 写道： > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > Hi, All. > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > I write a simple demo to illustrate my idea. Hope > > this > > > >>> helps. > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > Best, > > > >>> >>> > > > > > > Shengkai > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1 > > > >>> >>> > > > > > > > > > >>> >>> > > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> > > > 于2025年3月26日周三 > > > >>> >>> 15:54写道： > > > >>> >>> > > > > > > > > > >>> >>> > > > > > >>> I'm fine with a seperate SQL connector for > > metadata, > > > so > > > >>> >>> maybe > > > >>> >>> > we > > > >>> >>> > > > > could > > > >>> >>> > > > > > >> update the FLIP about our discussion? > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > >> Sorry, I've forgotten this part. Yeah, no matter > we > > > >>> choose > > > >>> >>> I'm > > > >>> >>> > > going > > > >>> >>> > > > > to > > > >>> >>> > > > > > >> update the FLIP. > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > >> G > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > >> On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi < > > > >>> >>> > > > > > gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >> wrote: > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > >>> Hi All, > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> I've also lack of the knowledge of PTF so I've > read > > > >>> just > > > >>> >>> the > > > >>> >>> > > > > motivation > > > >>> >>> > > > > > >>> part: > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> "The SQL 2016 standard introduced a way of > defining > > > >>> custom > > > >>> >>> SQL > > > >>> >>> > > > > > operators > > > >>> >>> > > > > > >>> defined by ISO/IEC 19075-7:2021 (Part 7: > > Polymorphic > > > >>> table > > > >>> >>> > > > > functions). > > > >>> >>> > > > > > >>> ~200 pages define how this new kind of function > can > > > >>> >>> consume and > > > >>> >>> > > > > produce > > > >>> >>> > > > > > >>> tables with various execution properties. > > > >>> >>> > > > > > >>> Unfortunately, this part of the standard is not > > > >>> publicly > > > >>> >>> > > > available." > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> Of course we can take a look at some examples but > > do > > > we > > > >>> >>> really > > > >>> >>> > > want > > > >>> >>> > > > > to > > > >>> >>> > > > > > >>> expose state data with this construct > > > >>> >>> > > > > > >>> which is described in ~200 pages and part of the > > > >>> standard > > > >>> >>> is > > > >>> >>> > not > > > >>> >>> > > > > > publicly > > > >>> >>> > > > > > >>> available? 🙂 > > > >>> >>> > > > > > >>> I mean the dataset is couple of rows and the > > use-case > > > >>> is > > > >>> >>> join > > > >>> >>> > > with > > > >>> >>> > > > > > >> another > > > >>> >>> > > > > > >>> table like with state data. > > > >>> >>> > > > > > >>> If somebody can give advantages I would buy that > > but > > > >>> from > > > >>> >>> my > > > >>> >>> > > > limited > > > >>> >>> > > > > > >>> understanding this would be an overkill here. > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> BR, > > > >>> >>> > > > > > >>> G > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>> On Wed, Mar 26, 2025 at 8:28 AM Gyula Fóra < > > > >>> >>> > gyula.f...@gmail.com > > > >>> >>> > > > > > > >>> >>> > > > > > wrote: > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>>> Hi Zakelly , Shengkai! > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>>> I don't know too much about PTFs, it would be > > > >>> interesting > > > >>> >>> to > > > >>> >>> > see > > > >>> >>> > > > how > > > >>> >>> > > > > > the > > > >>> >>> > > > > > >>>> usage would look in practice. > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>>> Do you have some mockup/example in mind how the > > PTF > > > >>> would > > > >>> >>> look > > > >>> >>> > > for > > > >>> >>> > > > > > >> example > > > >>> >>> > > > > > >>>> when want to: > > > >>> >>> > > > > > >>>> - Simply display/aggregate whats in the metadata > > > >>> >>> > > > > > >>>> - Join keyed state with some metadata columns > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>>> Thanks > > > >>> >>> > > > > > >>>> Gyula > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>>> On Wed, Mar 26, 2025 at 7:33 AM Zakelly Lan < > > > >>> >>> > > > zakelly....@gmail.com> > > > >>> >>> > > > > > >>>> wrote: > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>>>> Hi everyone, > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>> I'm fine with a seperate SQL connector for > > > metadata, > > > >>> so > > > >>> >>> maybe > > > >>> >>> > > we > > > >>> >>> > > > > > could > > > >>> >>> > > > > > >>>>> update the FLIP about our discussion? And > > Shengkai > > > >>> >>> provides a > > > >>> >>> > > PTF > > > >>> >>> > > > > > >>>>> implementation, does that also meet the > > > requirement? > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>> Best, > > > >>> >>> > > > > > >>>>> Zakelly > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>> On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi < > > > >>> >>> > > > > > >>>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>> wrote: > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>>> Hi All, > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>> @Zakelly: Gyula summarised it correctly what I > > > >>> meant so > > > >>> >>> > please > > > >>> >>> > > > > treat > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>> content as mine. > > > >>> >>> > > > > > >>>>>> As an addition I'm not against to add CLI at > > all, > > > >>> I'm > > > >>> >>> just > > > >>> >>> > > > stating > > > >>> >>> > > > > > >>>> that > > > >>> >>> > > > > > >>>>> in > > > >>> >>> > > > > > >>>>>> some cases like this, users would like to have > > > >>> >>> > > > > > >>>>>> a self-serving solution where they can provide > > SQL > > > >>> >>> > statements > > > >>> >>> > > > > which > > > >>> >>> > > > > > >>>> can > > > >>> >>> > > > > > >>>>>> trigger alerts automatically. > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>> My personal opinion is that CLI would be > > > beneficial > > > >>> for > > > >>> >>> > > several > > > >>> >>> > > > > > >>>> cases. A > > > >>> >>> > > > > > >>>>>> good example is when users want to restart job > > > >>> >>> > > > > > >>>>>> from specific Kafka offsets which are > persisted > > > in a > > > >>> >>> > > savepoint. > > > >>> >>> > > > > For > > > >>> >>> > > > > > >>>> such > > > >>> >>> > > > > > >>>>>> scenario users are more than happy since they > > > >>> >>> > > > > > >>>>>> expect manual intervention with full control. > So > > > >>> all in > > > >>> >>> all > > > >>> >>> > > one > > > >>> >>> > > > > can > > > >>> >>> > > > > > >>>> count > > > >>> >>> > > > > > >>>>>> on my +1 when CLI FLIP would come up... > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>> BR, > > > >>> >>> > > > > > >>>>>> G > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>> On Thu, Mar 20, 2025 at 8:20 AM Gyula Fóra < > > > >>> >>> > > > gyula.f...@gmail.com> > > > >>> >>> > > > > > >>>> wrote: > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>>>> Hi! > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>> @Zakelly Lan <zakelly....@gmail.com> > > > >>> >>> > > > > > >>>>>>> I think what Gabor means is that users want > to > > > have > > > >>> >>> > > predefined > > > >>> >>> > > > > SQL > > > >>> >>> > > > > > >>>>> scripts > > > >>> >>> > > > > > >>>>>>> to perform state analysis tasks to > > debug/identify > > > >>> >>> problems. > > > >>> >>> > > > > > >>>>>>> Such as write a SQL script that joins the > > > metadata > > > >>> >>> table > > > >>> >>> > with > > > >>> >>> > > > the > > > >>> >>> > > > > > >>>> state > > > >>> >>> > > > > > >>>>>>> and > > > >>> >>> > > > > > >>>>>>> do some analytics on it. > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>> If we have a meta table then the SQL script > > that > > > >>> can do > > > >>> >>> > this > > > >>> >>> > > is > > > >>> >>> > > > > > >> fixed > > > >>> >>> > > > > > >>>>> and > > > >>> >>> > > > > > >>>>>>> users can trigger this on demand by simply > > > >>> providing a > > > >>> >>> new > > > >>> >>> > > > > > >> savepoint > > > >>> >>> > > > > > >>>>> path. > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>> If we have a different mechanism to extract > > > >>> metadata > > > >>> >>> that > > > >>> >>> > is > > > >>> >>> > > > not > > > >>> >>> > > > > > >> SQL > > > >>> >>> > > > > > >>>>>>> native > > > >>> >>> > > > > > >>>>>>> then manual steps need to be executed and a > > > custom > > > >>> SQL > > > >>> >>> > script > > > >>> >>> > > > > would > > > >>> >>> > > > > > >>>> need > > > >>> >>> > > > > > >>>>>>> to > > > >>> >>> > > > > > >>>>>>> be written that adds the manually extracted > > > >>> metadata > > > >>> >>> into > > > >>> >>> > the > > > >>> >>> > > > > > >> script. > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>> Cheers, > > > >>> >>> > > > > > >>>>>>> Gyula > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>> On Thu, Mar 20, 2025 at 4:32 AM Zakelly Lan < > > > >>> >>> > > > > zakelly....@gmail.com > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>>>> Hi all, > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> Thanks for your answers! Getting everyone > > > aligned > > > >>> on > > > >>> >>> this > > > >>> >>> > > > topic > > > >>> >>> > > > > > >> is > > > >>> >>> > > > > > >>>>>>>> challenging, but it’s definitely worth the > > > effort > > > >>> >>> since it > > > >>> >>> > > > will > > > >>> >>> > > > > > >>>> help > > > >>> >>> > > > > > >>>>>>>> streamline things moving forward. > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> @Gabor are you saying that users are using > > some > > > >>> >>> scripts to > > > >>> >>> > > > > define > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>> SQL > > > >>> >>> > > > > > >>>>>>>> metadata connector and get the information, > > > >>> right? If > > > >>> >>> so, > > > >>> >>> > > > would > > > >>> >>> > > > > a > > > >>> >>> > > > > > >>>> CLI > > > >>> >>> > > > > > >>>>>>> tool > > > >>> >>> > > > > > >>>>>>>> be more convenient? It's easy to invoke and > > can > > > >>> get > > > >>> >>> the > > > >>> >>> > > result > > > >>> >>> > > > > > >>>>> swiftly. > > > >>> >>> > > > > > >>>>>>> And > > > >>> >>> > > > > > >>>>>>>> there should be some other systems to track > > the > > > >>> >>> checkpoint > > > >>> >>> > > > > > >> lineage > > > >>> >>> > > > > > >>>> and > > > >>> >>> > > > > > >>>>>>>> analyze if there are outliers in metadata > > (e.g. > > > >>> state > > > >>> >>> size > > > >>> >>> > > of > > > >>> >>> > > > > one > > > >>> >>> > > > > > >>>>>>> operator) > > > >>> >>> > > > > > >>>>>>>> right? Well, maybe I missed something so > > please > > > >>> >>> correct me > > > >>> >>> > > if > > > >>> >>> > > > > I'm > > > >>> >>> > > > > > >>>>> wrong. > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> I think the overall vision in Flink SQL is > to > > > >>> provide > > > >>> >>> a > > > >>> >>> > SQL > > > >>> >>> > > > > > >> native > > > >>> >>> > > > > > >>>>>>>>> environment where we can serve complex > > > use-cases > > > >>> >>> like you > > > >>> >>> > > > would > > > >>> >>> > > > > > >>>>> expect > > > >>> >>> > > > > > >>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>> regular database. > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> @Gyula Well, this is a good point. From the > > > >>> >>> perspective of > > > >>> >>> > > > > > >>>>> comprehensive > > > >>> >>> > > > > > >>>>>>>> SQL experience, I'd +1 for treating metadata > > as > > > >>> data. > > > >>> >>> > > > Although I > > > >>> >>> > > > > > >>>> doubt > > > >>> >>> > > > > > >>>>>>> if > > > >>> >>> > > > > > >>>>>>>> there is a need for processing metadata, I > > won't > > > >>> be > > > >>> >>> > against > > > >>> >>> > > a > > > >>> >>> > > > > > >>>> separate > > > >>> >>> > > > > > >>>>>>>> connector. > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> Regarding the CLI tool, I still think it’s > > worth > > > >>> >>> > > implementing. > > > >>> >>> > > > > > >>>> Such a > > > >>> >>> > > > > > >>>>>>> tool > > > >>> >>> > > > > > >>>>>>>> could provide savepoint information before > > > >>> resuming > > > >>> >>> from a > > > >>> >>> > > > > > >>>> savepoint, > > > >>> >>> > > > > > >>>>>>> which > > > >>> >>> > > > > > >>>>>>>> would enhance the user experience in > CLI-based > > > >>> >>> workflows. > > > >>> >>> > It > > > >>> >>> > > > > > >> would > > > >>> >>> > > > > > >>>> be > > > >>> >>> > > > > > >>>>>>> good > > > >>> >>> > > > > > >>>>>>>> if someone could implement this feature. We > > > >>> shouldn’t > > > >>> >>> > worry > > > >>> >>> > > > > about > > > >>> >>> > > > > > >>>>>>> whether > > > >>> >>> > > > > > >>>>>>>> this tool might be retired in the future. > > > >>> Regardless > > > >>> >>> of > > > >>> >>> > the > > > >>> >>> > > > > > >>>> SQL-based > > > >>> >>> > > > > > >>>>>>>> solution we eventually adopt, this > capability > > > will > > > >>> >>> remain > > > >>> >>> > > > > > >> essential > > > >>> >>> > > > > > >>>>> for > > > >>> >>> > > > > > >>>>>>> CLI > > > >>> >>> > > > > > >>>>>>>> users. This is another topic. > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>> Zakelly > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>> On Thu, Mar 20, 2025 at 10:37 AM Shengkai > > Fang < > > > >>> >>> > > > > > >> fskm...@gmail.com> > > > >>> >>> > > > > > >>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> Hi. > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> After reading the doc[1], I think Spark > > > provides > > > >>> a > > > >>> >>> > function > > > >>> >>> > > > for > > > >>> >>> > > > > > >>>>> users > > > >>> >>> > > > > > >>>>>>> to > > > >>> >>> > > > > > >>>>>>>>> consume the metadata from the savepoint. > In > > > >>> Flink > > > >>> >>> SQL, > > > >>> >>> > > > similar > > > >>> >>> > > > > > >>>>>>>>> functionality is implemented through > > > Polymorphic > > > >>> >>> Table > > > >>> >>> > > > > > >> Functions > > > >>> >>> > > > > > >>>>>>> (PTF) as > > > >>> >>> > > > > > >>>>>>>>> proposed in FLIP-440[2]. Below is a code > > > >>> example[3] > > > >>> >>> > > > > > >> illustrating > > > >>> >>> > > > > > >>>>> this > > > >>> >>> > > > > > >>>>>>>>> concept: > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> ``` > > > >>> >>> > > > > > >>>>>>>>> public static class ScalarArgsFunction > > > extends > > > >>> >>> > > > > > >>>>>>>>> TestProcessTableFunctionBase { > > > >>> >>> > > > > > >>>>>>>>> public void eval(Integer i, Boolean > > b) { > > > >>> >>> > > > > > >>>>>>>>> collectObjects(i, b); > > > >>> >>> > > > > > >>>>>>>>> } > > > >>> >>> > > > > > >>>>>>>>> } > > > >>> >>> > > > > > >>>>>>>>> ``` > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> ``` > > > >>> >>> > > > > > >>>>>>>>> INSERT INTO sink SELECT * FROM f(i => 42, b > > => > > > >>> >>> > CAST('TRUE' > > > >>> >>> > > AS > > > >>> >>> > > > > > >>>>>>> BOOLEAN)) > > > >>> >>> > > > > > >>>>>>>>> `` > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> So we can add a builtin function named > > > >>> >>> > > `read_state_metadata` > > > >>> >>> > > > to > > > >>> >>> > > > > > >>>> read > > > >>> >>> > > > > > >>>>>>>>> savepoint data. > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://docs.databricks.com/aws/en/structured-streaming/read-state?language=SQL > > > >>> >>> > > > > > >>>>>>>>> [2] > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=298781093 > > > >>> >>> > > > > > >>>>>>>>> [3] > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java#L140 > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> > > > 于2025年3月19日周三 > > > >>> >>> 18:37写道： > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> Hi All! > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> Thank you for the answers and concerns > from > > > >>> >>> everyone. > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> On the CLI vs State Metadata > Connector/Table > > > >>> >>> question I > > > >>> >>> > > > would > > > >>> >>> > > > > > >>>> also > > > >>> >>> > > > > > >>>>>>> like > > > >>> >>> > > > > > >>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>> step back a little and look at the bigger > > > >>> picture. > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> I think the overall vision in Flink SQL is > > to > > > >>> >>> provide a > > > >>> >>> > > SQL > > > >>> >>> > > > > > >>>> native > > > >>> >>> > > > > > >>>>>>>>>> environment where we can serve complex > > > use-cases > > > >>> >>> like > > > >>> >>> > you > > > >>> >>> > > > > > >> would > > > >>> >>> > > > > > >>>>>>> expect > > > >>> >>> > > > > > >>>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>>> regular database. > > > >>> >>> > > > > > >>>>>>>>>> Most features, developments in the recent > > > years > > > >>> have > > > >>> >>> > gone > > > >>> >>> > > > > > >> this > > > >>> >>> > > > > > >>>>> way. > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> The State Metadata Table would be a > natural > > > and > > > >>> >>> > > > > > >> straightforward > > > >>> >>> > > > > > >>>>> fit > > > >>> >>> > > > > > >>>>>>>> here. > > > >>> >>> > > > > > >>>>>>>>>> So from my side, +1 for that. > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> However I could understand if we are not > > ready > > > >>> to > > > >>> >>> add a > > > >>> >>> > > new > > > >>> >>> > > > > > >>>>>>>>>> connector/format due to maintenance > concerns > > > >>> (and in > > > >>> >>> > > general > > > >>> >>> > > > > > >>>>> concern > > > >>> >>> > > > > > >>>>>>>>> about > > > >>> >>> > > > > > >>>>>>>>>> the design). > > > >>> >>> > > > > > >>>>>>>>>> If that's the issue then we should spend > > more > > > >>> time > > > >>> >>> on > > > >>> >>> > the > > > >>> >>> > > > > > >>>> design > > > >>> >>> > > > > > >>>>> to > > > >>> >>> > > > > > >>>>>>> get > > > >>> >>> > > > > > >>>>>>>>>> comfortable with the approach and seek > > > feedback > > > >>> >>> from the > > > >>> >>> > > > > > >> wider > > > >>> >>> > > > > > >>>>>>>> community > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> I am -1 for the CLI/tooling approach as > that > > > >>> will > > > >>> >>> not > > > >>> >>> > > > provide > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>>> featureset we are looking for that is not > > > >>> already > > > >>> >>> > covered > > > >>> >>> > > by > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>> Java > > > >>> >>> > > > > > >>>>>>>>>> connector. And that approach would come > with > > > the > > > >>> >>> same > > > >>> >>> > > > > > >>>> maintenance > > > >>> >>> > > > > > >>>>>>>>>> implications. > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> Cheers > > > >>> >>> > > > > > >>>>>>>>>> Gyula > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> On Wed, Mar 19, 2025 at 11:24 AM Gabor > > > Somogyi < > > > >>> >>> > > > > > >>>>>>>>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> Hi Zaklely, Shengkai > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> Several topics are going on so adding > gist > > > >>> answers > > > >>> >>> to > > > >>> >>> > > them. > > > >>> >>> > > > > > >>>> When > > > >>> >>> > > > > > >>>>>>> some > > > >>> >>> > > > > > >>>>>>>>>> topic > > > >>> >>> > > > > > >>>>>>>>>>> is not touched please highlight it. > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> @Shengkai: I've read through all the > > previous > > > >>> FLIPs > > > >>> >>> > > related > > > >>> >>> > > > > > >>>>>>> catalogs > > > >>> >>> > > > > > >>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>> if > > > >>> >>> > > > > > >>>>>>>>>>> we would like to keep the concepts there > > > >>> >>> > > > > > >>>>>>>>>>> then one-to-one mapping relationship > > between > > > >>> >>> savepoint > > > >>> >>> > > and > > > >>> >>> > > > > > >>>>> catalog > > > >>> >>> > > > > > >>>>>>>> is a > > > >>> >>> > > > > > >>>>>>>>>>> reasonable direction. In short I'm happy > > that > > > >>> >>> > > > > > >>>>>>>>>>> you've highlighted this and agree as a > > whole. > > > >>> I've > > > >>> >>> > > written > > > >>> >>> > > > > > >> it > > > >>> >>> > > > > > >>>>> down > > > >>> >>> > > > > > >>>>>>>>>>> previously, just want to double confirm > > that > > > >>> state > > > >>> >>> > > catalog > > > >>> >>> > > > > > >> is > > > >>> >>> > > > > > >>>>>>>>>>> essential and planned. When we reach this > > > point > > > >>> >>> then > > > >>> >>> > your > > > >>> >>> > > > > > >>>> input > > > >>> >>> > > > > > >>>>> is > > > >>> >>> > > > > > >>>>>>>> more > > > >>> >>> > > > > > >>>>>>>>>>> than welcome. > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> @Zakelly: We've tried the CLI and > separate > > > >>> library > > > >>> >>> > > > > > >> approaches > > > >>> >>> > > > > > >>>>> with > > > >>> >>> > > > > > >>>>>>>>> users > > > >>> >>> > > > > > >>>>>>>>>>> already and these are not something which > > is > > > >>> >>> welcome > > > >>> >>> > > > > > >> because > > > >>> >>> > > > > > >>>> of > > > >>> >>> > > > > > >>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>> following: > > > >>> >>> > > > > > >>>>>>>>>>> * Users want to have automated tasks and > > not > > > >>> manual > > > >>> >>> > > > > > >>>> CLI/library > > > >>> >>> > > > > > >>>>>>>> output > > > >>> >>> > > > > > >>>>>>>>>>> parsing. This can be hacked around but > our > > > >>> >>> experience > > > >>> >>> > is > > > >>> >>> > > > > > >>>>> negative > > > >>> >>> > > > > > >>>>>>> on > > > >>> >>> > > > > > >>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>> because it's just brittle. > > > >>> >>> > > > > > >>>>>>>>>>> * From development perspective It's way > > much > > > >>> bigger > > > >>> >>> > > effort > > > >>> >>> > > > > > >>>> than > > > >>> >>> > > > > > >>>>> a > > > >>> >>> > > > > > >>>>>>>>>> connector > > > >>> >>> > > > > > >>>>>>>>>>> (hard to test, packaging/version handling > > is > > > >>> and > > > >>> >>> extra > > > >>> >>> > > > > > >> layer > > > >>> >>> > > > > > >>>> of > > > >>> >>> > > > > > >>>>>>>>>> complexity, > > > >>> >>> > > > > > >>>>>>>>>>> external FS authentication is pain for > > users, > > > >>> >>> expecting > > > >>> >>> > > > > > >> them > > > >>> >>> > > > > > >>>> to > > > >>> >>> > > > > > >>>>>>>>> download > > > >>> >>> > > > > > >>>>>>>>>>> savepoints also) > > > >>> >>> > > > > > >>>>>>>>>>> * Purely personal opinion but if we would > > > find > > > >>> >>> better > > > >>> >>> > > ways > > > >>> >>> > > > > > >>>> later > > > >>> >>> > > > > > >>>>>>> then > > > >>> >>> > > > > > >>>>>>>>>>> retire a CLI is not more lightweight than > > > >>> retire a > > > >>> >>> > > > > > >> connector > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> It would be great if you give some > > examples > > > >>> on how > > > >>> >>> > user > > > >>> >>> > > > > > >>>> could > > > >>> >>> > > > > > >>>>>>>>> leverage > > > >>> >>> > > > > > >>>>>>>>>>> the separate connector to process the > > > metadata. > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> The most simplest cases: > > > >>> >>> > > > > > >>>>>>>>>>> * give me the overgroving state uids > > > >>> >>> > > > > > >>>>>>>>>>> * give me the not known (new or renamed) > > > state > > > >>> uids > > > >>> >>> > > > > > >>>>>>>>>>> * give me the state uids where state size > > > >>> >>> drastically > > > >>> >>> > > > > > >> dropped > > > >>> >>> > > > > > >>>>>>> compare > > > >>> >>> > > > > > >>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>> previous savepoint (accidental state > loss) > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> Since it was mentioned: as a general > > offtopic > > > >>> >>> teaser, > > > >>> >>> > > yeah > > > >>> >>> > > > > > >> it > > > >>> >>> > > > > > >>>>>>> would > > > >>> >>> > > > > > >>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>> good > > > >>> >>> > > > > > >>>>>>>>>>> to have some sort of checkpoint/savepoint > > > >>> lineage > > > >>> >>> or > > > >>> >>> > > > > > >> however > > > >>> >>> > > > > > >>>> we > > > >>> >>> > > > > > >>>>>>> call > > > >>> >>> > > > > > >>>>>>>>> it. > > > >>> >>> > > > > > >>>>>>>>>>> Since we've not yet reached this point > > there > > > >>> are no > > > >>> >>> > > > > > >> technical > > > >>> >>> > > > > > >>>>>>>> details, > > > >>> >>> > > > > > >>>>>>>>>> it's > > > >>> >>> > > > > > >>>>>>>>>>> more like a vision. It's a common pattern > > > that > > > >>> >>> > > > > > >>>>>>>>>>> jobs are physically running but somehow > the > > > >>> state > > > >>> >>> > > > > > >> processing > > > >>> >>> > > > > > >>>> is > > > >>> >>> > > > > > >>>>>>> stuck > > > >>> >>> > > > > > >>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>> it would be good to add some way to find > it > > > out > > > >>> >>> > > > > > >>>> automatically. > > > >>> >>> > > > > > >>>>>>>>>>> The important saying here is automation > and > > > not > > > >>> >>> manual > > > >>> >>> > > > > > >>>>> evaluation > > > >>> >>> > > > > > >>>>>>>> since > > > >>> >>> > > > > > >>>>>>>>>>> handling 10k+ jobs is just not allowing > > that. > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> On Wed, Mar 19, 2025 at 6:46 AM Shengkai > > > Fang < > > > >>> >>> > > > > > >>>>> fskm...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> Hi, All. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> About State Catalog, I want to share > more > > > >>> thoughts > > > >>> >>> > about > > > >>> >>> > > > > > >>>> this. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> In the initial design concept, I > > understood > > > >>> that a > > > >>> >>> > > > > > >>>> savepoint > > > >>> >>> > > > > > >>>>>>> and a > > > >>> >>> > > > > > >>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>> catalog have a one-to-one mapping > > > >>> relationship. > > > >>> >>> Each > > > >>> >>> > > > > > >>>> operator > > > >>> >>> > > > > > >>>>>>>>>> corresponds > > > >>> >>> > > > > > >>>>>>>>>>>> to a database, and the state of each > > > operator > > > >>> is > > > >>> >>> > > > > > >>>> represented > > > >>> >>> > > > > > >>>>> as > > > >>> >>> > > > > > >>>>>>>>>>> individual > > > >>> >>> > > > > > >>>>>>>>>>>> tables. The rationale behind this design > > is: > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> *State Diversity*: An operator may > involve > > > >>> >>> multiple > > > >>> >>> > > types > > > >>> >>> > > > > > >>>> of > > > >>> >>> > > > > > >>>>>>>> states. > > > >>> >>> > > > > > >>>>>>>>>> For > > > >>> >>> > > > > > >>>>>>>>>>>> example, in our VVR design, a > "multi-join" > > > >>> >>> operator > > > >>> >>> > uses > > > >>> >>> > > > > > >>>> keyed > > > >>> >>> > > > > > >>>>>>>> states > > > >>> >>> > > > > > >>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>> two input streams and a broadcast state > > for > > > >>> the > > > >>> >>> third > > > >>> >>> > > > > > >>>> stream. > > > >>> >>> > > > > > >>>>>>> This > > > >>> >>> > > > > > >>>>>>>>>> makes > > > >>> >>> > > > > > >>>>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>> challenging to represent all states of > an > > > >>> operator > > > >>> >>> > > > > > >> within a > > > >>> >>> > > > > > >>>>>>> single > > > >>> >>> > > > > > >>>>>>>>>> table. > > > >>> >>> > > > > > >>>>>>>>>>>> *Scalability*: Internally, an operator > > might > > > >>> have > > > >>> >>> > > > > > >> multiple > > > >>> >>> > > > > > >>>>> keyed > > > >>> >>> > > > > > >>>>>>>>> states > > > >>> >>> > > > > > >>>>>>>>>>>> (e.g., value state and list state). > > However, > > > >>> large > > > >>> >>> > list > > > >>> >>> > > > > > >>>> states > > > >>> >>> > > > > > >>>>>>> may > > > >>> >>> > > > > > >>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>> fit > > > >>> >>> > > > > > >>>>>>>>>>>> entirely in memory. To address this, we > > > >>> recommend > > > >>> >>> > > > > > >>>> implementing > > > >>> >>> > > > > > >>>>>>> each > > > >>> >>> > > > > > >>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>> as a separate table. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> To resolve the loosely coupled > > relationships > > > >>> >>> between > > > >>> >>> > > > > > >>>> operator > > > >>> >>> > > > > > >>>>>>>> states, > > > >>> >>> > > > > > >>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>> propose embedding predefined views > within > > > the > > > >>> >>> catalog. > > > >>> >>> > > > > > >>>> These > > > >>> >>> > > > > > >>>>>>> views > > > >>> >>> > > > > > >>>>>>>>>>> simplify > > > >>> >>> > > > > > >>>>>>>>>>>> user understanding of operator > > > >>> implementations and > > > >>> >>> > > > > > >> provide > > > >>> >>> > > > > > >>>> a > > > >>> >>> > > > > > >>>>>>> more > > > >>> >>> > > > > > >>>>>>>>>>> intuitive > > > >>> >>> > > > > > >>>>>>>>>>>> perspective. For instance, a join > operator > > > may > > > >>> >>> have > > > >>> >>> > > > > > >>>> multiple > > > >>> >>> > > > > > >>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>> implementations (depending on whether > the > > > >>> join key > > > >>> >>> > > > > > >> includes > > > >>> >>> > > > > > >>>>>>> unique > > > >>> >>> > > > > > >>>>>>>>>>>> attributes), but users primarily care > > about > > > >>> the > > > >>> >>> data > > > >>> >>> > > > > > >>>>> associated > > > >>> >>> > > > > > >>>>>>>> with > > > >>> >>> > > > > > >>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>>> specific join key across input streams. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> Returning to the one-to-one mapping > > between > > > >>> >>> savepoints > > > >>> >>> > > > > > >> and > > > >>> >>> > > > > > >>>>>>>> catalogs, > > > >>> >>> > > > > > >>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>> aim > > > >>> >>> > > > > > >>>>>>>>>>>> to manage multiple user state catalogs > > > >>> through a > > > >>> >>> > catalog > > > >>> >>> > > > > > >>>>> store. > > > >>> >>> > > > > > >>>>>>>> When > > > >>> >>> > > > > > >>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>> user > > > >>> >>> > > > > > >>>>>>>>>>>> triggers a savepoint for a job on the > > > >>> platform: > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> 1. The platform sends a REST request to > > the > > > >>> >>> > JobManager. > > > >>> >>> > > > > > >>>>>>>>>>>> 2. Simultaneously, it registers a new > > state > > > >>> >>> catalog in > > > >>> >>> > > > > > >> the > > > >>> >>> > > > > > >>>>>>> catalog > > > >>> >>> > > > > > >>>>>>>>>> store, > > > >>> >>> > > > > > >>>>>>>>>>>> enabling immediate analysis of state > data > > on > > > >>> the > > > >>> >>> > > > > > >> platform. > > > >>> >>> > > > > > >>>>>>>>>>>> 3. Deleting a savepoint would also > trigger > > > the > > > >>> >>> removal > > > >>> >>> > > of > > > >>> >>> > > > > > >>>> its > > > >>> >>> > > > > > >>>>>>>>>> associated > > > >>> >>> > > > > > >>>>>>>>>>>> catalog. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> This vision assumes that states are > > > >>> >>> self-describing or > > > >>> >>> > > > > > >>>> that a > > > >>> >>> > > > > > >>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>> metaservice is introduced to analyze > > > savepoint > > > >>> >>> > > > > > >> structures. > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> How can users create logic to identify > > > >>> >>> differences > > > >>> >>> > > > > > >>>> between > > > >>> >>> > > > > > >>>>>>>> multiple > > > >>> >>> > > > > > >>>>>>>>>>>> savepoints? > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> Since savepoints and state catalogs are > > > >>> one-to-one > > > >>> >>> > > > > > >> mapped, > > > >>> >>> > > > > > >>>>> users > > > >>> >>> > > > > > >>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>> query > > > >>> >>> > > > > > >>>>>>>>>>>> metadata via their respective catalogs. > > For > > > >>> >>> example: > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> 1. > > > >>> >>> > > > > > >>>>> > > > >>> >>> `savepoint-${id}`.`system`.`metadata_table`.`<operator-name>` > > > >>> >>> > > > > > >>>>>>>>>> provides > > > >>> >>> > > > > > >>>>>>>>>>>> operator-specific metadata (e.g., state > > > size, > > > >>> >>> type). > > > >>> >>> > > > > > >>>>>>>>>>>> 2. Comparing metadata tables (e.g., > schema > > > >>> >>> versions, > > > >>> >>> > > > > > >> state > > > >>> >>> > > > > > >>>>> entry > > > >>> >>> > > > > > >>>>>>>>>> counts) > > > >>> >>> > > > > > >>>>>>>>>>>> across catalogs reveals structural or > > > >>> quantitative > > > >>> >>> > > > > > >>>>> differences. > > > >>> >>> > > > > > >>>>>>>>>>>> 3. For deeper analysis, users could > write > > > SQL > > > >>> >>> queries > > > >>> >>> > to > > > >>> >>> > > > > > >>>>> compare > > > >>> >>> > > > > > >>>>>>>>>> specific > > > >>> >>> > > > > > >>>>>>>>>>>> state partitions or leverage the > > metaservice > > > >>> to > > > >>> >>> track > > > >>> >>> > > > > > >> state > > > >>> >>> > > > > > >>>>>>>> evolution > > > >>> >>> > > > > > >>>>>>>>>>>> (e.g., added/removed operators, modified > > > state > > > >>> >>> > > > > > >>>>> configurations). > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> If we plan to introduce a state catalog > in > > > the > > > >>> >>> > future, I > > > >>> >>> > > > > > >>>> would > > > >>> >>> > > > > > >>>>>>> lean > > > >>> >>> > > > > > >>>>>>>>>>> toward > > > >>> >>> > > > > > >>>>>>>>>>>> using metadata tables. If a utility tool > > can > > > >>> >>> address > > > >>> >>> > the > > > >>> >>> > > > > > >>>>>>> challenges > > > >>> >>> > > > > > >>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>> face, could we avoid introducing an > > > additional > > > >>> >>> > > connector? > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> > > > >>> 于2025年3月17日周一 > > > >>> >>> > > 20:25写道： > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> Hi All! > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> Without going into too much detail here > > are > > > >>> my 2 > > > >>> >>> > cents > > > >>> >>> > > > > > >>>>>>> regarding > > > >>> >>> > > > > > >>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> virtual column / catalog metadata / > table > > > >>> >>> (connector) > > > >>> >>> > > > > > >>>>>>> discussion > > > >>> >>> > > > > > >>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> State metadata. > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> State metadata such as the types of > > states, > > > >>> their > > > >>> >>> > > > > > >>>>> properties, > > > >>> >>> > > > > > >>>>>>>>> names, > > > >>> >>> > > > > > >>>>>>>>>>>> sizes > > > >>> >>> > > > > > >>>>>>>>>>>>> etc are all valuable information that > can > > > be > > > >>> >>> used to > > > >>> >>> > > > > > >>>> enrich > > > >>> >>> > > > > > >>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> computations we do on state. > > > >>> >>> > > > > > >>>>>>>>>>>>> We can either analyze it standalone > (such > > > as > > > >>> >>> discover > > > >>> >>> > > > > > >>>>>>> anomalies, > > > >>> >>> > > > > > >>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>> large > > > >>> >>> > > > > > >>>>>>>>>>>>> jobs with many states), across multiple > > > >>> >>> savepoints > > > >>> >>> > > > > > >>>> (discover > > > >>> >>> > > > > > >>>>>>> how > > > >>> >>> > > > > > >>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>> changed over time) or by joining it > with > > > >>> keyed or > > > >>> >>> > > > > > >>>> non-keyed > > > >>> >>> > > > > > >>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>> data > > > >>> >>> > > > > > >>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>> serve more complex queries on the > state. > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> The only solution that seems to serve > all > > > >>> these > > > >>> >>> > > > > > >> use-cases > > > >>> >>> > > > > > >>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>> requirements > > > >>> >>> > > > > > >>>>>>>>>>>>> in a straightforward and SQL canonical > > way > > > >>> is to > > > >>> >>> > simply > > > >>> >>> > > > > > >>>>> expose > > > >>> >>> > > > > > >>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>> metadata as a separate table. This is a > > > >>> metadata > > > >>> >>> > table > > > >>> >>> > > > > > >>>> but > > > >>> >>> > > > > > >>>>> you > > > >>> >>> > > > > > >>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>> also > > > >>> >>> > > > > > >>>>>>>>>>>>> think of it as data table, it makes no > > > >>> practical > > > >>> >>> > > > > > >>>> difference > > > >>> >>> > > > > > >>>>>>> here. > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> Once we have a catalog later, the > catalog > > > can > > > >>> >>> offer > > > >>> >>> > > > > > >> this > > > >>> >>> > > > > > >>>>> table > > > >>> >>> > > > > > >>>>>>>> out > > > >>> >>> > > > > > >>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> box, the same way databases provide > > > metadata > > > >>> >>> tables. > > > >>> >>> > > > > > >> For > > > >>> >>> > > > > > >>>>> this > > > >>> >>> > > > > > >>>>>>> to > > > >>> >>> > > > > > >>>>>>>>> work > > > >>> >>> > > > > > >>>>>>>>>>>>> however we need another, simpler > > connector > > > >>> that > > > >>> >>> > creates > > > >>> >>> > > > > > >>>> this > > > >>> >>> > > > > > >>>>>>>> table. > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> +1 for state metadata as a separate > > > >>> >>> connector/table, > > > >>> >>> > > > > > >>>> instead > > > >>> >>> > > > > > >>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>> adding > > > >>> >>> > > > > > >>>>>>>>>>>>> virtual columns and adhoc catalog > > metadata > > > >>> that > > > >>> >>> is > > > >>> >>> > hard > > > >>> >>> > > > > > >>>> to > > > >>> >>> > > > > > >>>>> use > > > >>> >>> > > > > > >>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>>>> large > > > >>> >>> > > > > > >>>>>>>>>>>>> number of queries. > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> Cheers, > > > >>> >>> > > > > > >>>>>>>>>>>>> Gyula > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> On Mon, Mar 17, 2025 at 12:44 PM Gabor > > > >>> Somogyi < > > > >>> >>> > > > > > >>>>>>>>>>>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> 1. State TTL for Value Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> I’m planning on adding this, and we > may > > > >>> >>> collaborate > > > >>> >>> > > > > > >>>> on > > > >>> >>> > > > > > >>>>> it > > > >>> >>> > > > > > >>>>>>> in > > > >>> >>> > > > > > >>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> future. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> +1 on this, just ping me. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> After some code digging and POC all I > > can > > > >>> say > > > >>> >>> that > > > >>> >>> > > > > > >> with > > > >>> >>> > > > > > >>>>>>> heavy > > > >>> >>> > > > > > >>>>>>>>>> effort > > > >>> >>> > > > > > >>>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>> maybe add such changes that we're able > > to > > > >>> show > > > >>> >>> > > > > > >> metadata > > > >>> >>> > > > > > >>>>> of a > > > >>> >>> > > > > > >>>>>>>>>>> savepoint > > > >>> >>> > > > > > >>>>>>>>>>>>> from > > > >>> >>> > > > > > >>>>>>>>>>>>>> catalog. > > > >>> >>> > > > > > >>>>>>>>>>>>>> I'm not against that but from user > > > >>> perspective > > > >>> >>> this > > > >>> >>> > > > > > >> has > > > >>> >>> > > > > > >>>>>>> limited > > > >>> >>> > > > > > >>>>>>>>>>> value, > > > >>> >>> > > > > > >>>>>>>>>>>>> let > > > >>> >>> > > > > > >>>>>>>>>>>>>> me explain why. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> From high level perspective I see the > > > >>> following > > > >>> >>> > > > > > >> which I > > > >>> >>> > > > > > >>>>> see > > > >>> >>> > > > > > >>>>>>>>>> agreement > > > >>> >>> > > > > > >>>>>>>>>>>> on: > > > >>> >>> > > > > > >>>>>>>>>>>>>> * We should have a catalog which is > > > >>> >>> representing one > > > >>> >>> > > > > > >> or > > > >>> >>> > > > > > >>>>> more > > > >>> >>> > > > > > >>>>>>>> jobs > > > >>> >>> > > > > > >>>>>>>>>>>>> savepoint > > > >>> >>> > > > > > >>>>>>>>>>>>>> data set (future plan) > > > >>> >>> > > > > > >>>>>>>>>>>>>> * Savepoints should be able to be > > > >>> registered in > > > >>> >>> the > > > >>> >>> > > > > > >>>>> catalog > > > >>> >>> > > > > > >>>>>>>> which > > > >>> >>> > > > > > >>>>>>>>>> are > > > >>> >>> > > > > > >>>>>>>>>>>>> then > > > >>> >>> > > > > > >>>>>>>>>>>>>> databases (future plan) > > > >>> >>> > > > > > >>>>>>>>>>>>>> * There must be a possiblity to create > > > >>> tables > > > >>> >>> from > > > >>> >>> > > > > > >>>>> databases > > > >>> >>> > > > > > >>>>>>>>> where > > > >>> >>> > > > > > >>>>>>>>>>>> users > > > >>> >>> > > > > > >>>>>>>>>>>>>> can read state data (exists already) > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> In terms of metadata, If I understand > > > >>> correctly > > > >>> >>> then > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>> suggested > > > >>> >>> > > > > > >>>>>>>>>>>>> approach > > > >>> >>> > > > > > >>>>>>>>>>>>>> would be to access > > > >>> >>> > > > > > >>>>>>>>>>>>>> it from the catalog describe command, > > > right? > > > >>> >>> Adding > > > >>> >>> > > > > > >>>> that > > > >>> >>> > > > > > >>>>>>> info > > > >>> >>> > > > > > >>>>>>>>> when > > > >>> >>> > > > > > >>>>>>>>>>>>> specific > > > >>> >>> > > > > > >>>>>>>>>>>>>> database describe command > > > >>> >>> > > > > > >>>>>>>>>>>>>> is executed could be done. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> The question is for instance how can > > users > > > >>> >>> create > > > >>> >>> > > > > > >> such > > > >>> >>> > > > > > >>>> a > > > >>> >>> > > > > > >>>>>>> logic > > > >>> >>> > > > > > >>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>> tells > > > >>> >>> > > > > > >>>>>>>>>>>>>> them what is > > > >>> >>> > > > > > >>>>>>>>>>>>>> the difference between multiple > > > savepoints? > > > >>> >>> > > > > > >>>>>>>>>>>>>> Just to give some examples: > > > >>> >>> > > > > > >>>>>>>>>>>>>> * per operator size changes between > > > >>> savepoints > > > >>> >>> > > > > > >>>>>>>>>>>>>> * show values from operator data where > > > state > > > >>> >>> size > > > >>> >>> > > > > > >>>> reaches > > > >>> >>> > > > > > >>>>> a > > > >>> >>> > > > > > >>>>>>>>>> boundary > > > >>> >>> > > > > > >>>>>>>>>>>>>> * in general "find which checkpoint > > ruined > > > >>> >>> things" > > > >>> >>> > is > > > >>> >>> > > > > > >>>>> quite > > > >>> >>> > > > > > >>>>>>>>> common > > > >>> >>> > > > > > >>>>>>>>>>>>> pattern > > > >>> >>> > > > > > >>>>>>>>>>>>>> What I would like to highlight here is > > > that > > > >>> from > > > >>> >>> > > > > > >> Flink > > > >>> >>> > > > > > >>>>>>> point of > > > >>> >>> > > > > > >>>>>>>>>> view > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> metadata can be > > > >>> >>> > > > > > >>>>>>>>>>>>>> considered as a static side output > > > >>> information > > > >>> >>> but > > > >>> >>> > > > > > >> for > > > >>> >>> > > > > > >>>>> users > > > >>> >>> > > > > > >>>>>>>>> these > > > >>> >>> > > > > > >>>>>>>>>>>> values > > > >>> >>> > > > > > >>>>>>>>>>>>>> are actual real data > > > >>> >>> > > > > > >>>>>>>>>>>>>> where logic is planned to build > around. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> The metadata is more like one-time > > > >>> information > > > >>> >>> > > > > > >>>> instead > > > >>> >>> > > > > > >>>>> of > > > >>> >>> > > > > > >>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>> streaming > > > >>> >>> > > > > > >>>>>>>>>>>>>> data that changes all > > > >>> >>> > > > > > >>>>>>>>>>>>>> the time, so a single connector seems > to > > > be > > > >>> an > > > >>> >>> > > > > > >>>> overkill. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> State data is also static within a > > > >>> savepoint and > > > >>> >>> > > > > > >> that's > > > >>> >>> > > > > > >>>>> the > > > >>> >>> > > > > > >>>>>>>>> reason > > > >>> >>> > > > > > >>>>>>>>>>> why > > > >>> >>> > > > > > >>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> state processor API is working in > batch > > > >>> mode. > > > >>> >>> > > > > > >>>>>>>>>>>>>> When we handle multiple checkpoints > in a > > > >>> >>> streaming > > > >>> >>> > > > > > >>>> fashion > > > >>> >>> > > > > > >>>>>>> then > > > >>> >>> > > > > > >>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>> viewed from another angle. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> We can come up with more lightweight > > > >>> solution > > > >>> >>> other > > > >>> >>> > > > > > >>>> than a > > > >>> >>> > > > > > >>>>>>> new > > > >>> >>> > > > > > >>>>>>>>>>>> connector > > > >>> >>> > > > > > >>>>>>>>>>>>>> but enforcing users to parse the > catalog > > > >>> >>> > > > > > >>>>>>>>>>>>>> describe command output in order to > > > compare > > > >>> >>> multiple > > > >>> >>> > > > > > >>>>>>> savepoints > > > >>> >>> > > > > > >>>>>>>>>>> doesn't > > > >>> >>> > > > > > >>>>>>>>>>>>>> sound smooth user experience. > > > >>> >>> > > > > > >>>>>>>>>>>>>> Honestly I've no other idea how > exposing > > > >>> >>> metadata as > > > >>> >>> > > > > > >>>> real > > > >>> >>> > > > > > >>>>>>> user > > > >>> >>> > > > > > >>>>>>>>> data > > > >>> >>> > > > > > >>>>>>>>>>> so > > > >>> >>> > > > > > >>>>>>>>>>>>>> waiting on other approaches. > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> On Thu, Mar 13, 2025 at 2:44 AM > Shengkai > > > >>> Fang < > > > >>> >>> > > > > > >>>>>>>> fskm...@gmail.com > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Looking forward to hearing the good > > news! > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Gabor Somogyi < > > gabor.g.somo...@gmail.com > > > > > > > >>> >>> > > > > > >>>> 于2025年3月12日周三 > > > >>> >>> > > > > > >>>>>>>>> 22:24写道： > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> Thanks for both the valuable input! > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> Let me take a closer look at the > > > >>> suggestions, > > > >>> >>> > > > > > >> like > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>> Catalog > > > >>> >>> > > > > > >>>>>>>>>>>>>>> capabilities > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> and possibility of embedding > > > >>> TypeInformation > > > >>> >>> or > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> StateDescriptor metadata directly > into > > > >>> the raw > > > >>> >>> > > > > > >>>> state > > > >>> >>> > > > > > >>>>>>>> files... > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 8:17 AM > > Shengkai > > > >>> Fang > > > >>> >>> < > > > >>> >>> > > > > > >>>>>>>>>> fskm...@gmail.com > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> Thanks for Zakelly's clarification. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> +1 to delay the discussion about > > this. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata > Column > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> I’d like to share my perspective on > > the > > > >>> State > > > >>> >>> > > > > > >>>>> Catalog > > > >>> >>> > > > > > >>>>>>>>>> proposal. > > > >>> >>> > > > > > >>>>>>>>>>>>> While > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> introducing this capability is > > > >>> beneficial, > > > >>> >>> > > > > > >> there > > > >>> >>> > > > > > >>>> is > > > >>> >>> > > > > > >>>>> a > > > >>> >>> > > > > > >>>>>>>>>> blocker: > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> current > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> StateBackend architecture does not > > > permit > > > >>> >>> > > > > > >>>> operators > > > >>> >>> > > > > > >>>>> to > > > >>> >>> > > > > > >>>>>>>>> encode > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> TypeInformation into the state—it > > only > > > >>> >>> > > > > > >> preserves > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>>>> Serializer. > > > >>> >>> > > > > > >>>>>>>>>>>>> This > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> limitation creates an asymmetry, as > > > >>> operators > > > >>> >>> > > > > > >>>> alone > > > >>> >>> > > > > > >>>>>>>> retain > > > >>> >>> > > > > > >>>>>>>>>>>>> knowledge > > > >>> >>> > > > > > >>>>>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> data structure’s schema. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> To address this, I suggest allowing > > > >>> operators > > > >>> >>> > > > > > >> to > > > >>> >>> > > > > > >>>>> embed > > > >>> >>> > > > > > >>>>>>>>>>>>>> TypeInformation > > > >>> >>> > > > > > >>>>>>>>>>>>>>> or > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> StateDescriptor metadata directly > > into > > > >>> the > > > >>> >>> raw > > > >>> >>> > > > > > >>>> state > > > >>> >>> > > > > > >>>>>>>> files. > > > >>> >>> > > > > > >>>>>>>>>>> Such > > > >>> >>> > > > > > >>>>>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>>>>>> design > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> would enable the Catalog to: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> 1. Parse state files and > > > programmatically > > > >>> >>> > > > > > >> derive > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>> schema > > > >>> >>> > > > > > >>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> structural > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> guarantees for each state. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> 2. Leverage existing Flink Table > > > >>> utilities, > > > >>> >>> > > > > > >> such > > > >>> >>> > > > > > >>>> as > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> LegacyTypeInfoDataTypeConverter (in > > > >>> >>> > > > > > >>>>>>>>>>>>>>> org.apache.flink.table.types.utils), > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> bridge TypeInformation and DataType > > > >>> >>> > > > > > >> conversions. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> If we can not store the > > TypeInformation > > > >>> or > > > >>> >>> > > > > > >>>>>>>> StateDescriptor > > > >>> >>> > > > > > >>>>>>>>>> into > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> raw > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> state files, I am +1 for this FLIP > to > > > use > > > >>> >>> > > > > > >>>> metadata > > > >>> >>> > > > > > >>>>>>> column > > > >>> >>> > > > > > >>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>> retrieve > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> information. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> Zakelly Lan <zakelly....@gmail.com > > > > > >>> >>> > > > > > >>>> 于2025年3月12日周三 > > > >>> >>> > > > > > >>>>>>>> 12:43写道： > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Hi Gabor and Shengkai, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Thanks for sharing your thoughts! > > This > > > >>> is a > > > >>> >>> > > > > > >>>> long > > > >>> >>> > > > > > >>>>>>>>> discussion > > > >>> >>> > > > > > >>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>> sorry > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the late reply (I'm busy catching > up > > > >>> with > > > >>> >>> > > > > > >>>> release > > > >>> >>> > > > > > >>>>>>> 2.0 > > > >>> >>> > > > > > >>>>>>>>> these > > > >>> >>> > > > > > >>>>>>>>>>>>> days). > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Let me first clarify your thoughts > > to > > > >>> ensure > > > >>> >>> > > > > > >> I > > > >>> >>> > > > > > >>>>>>>> understand > > > >>> >>> > > > > > >>>>>>>>>>>>>> correctly. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> IIUC, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> there is no persistent > configuration > > > for > > > >>> >>> > > > > > >> state > > > >>> >>> > > > > > >>>> TTL > > > >>> >>> > > > > > >>>>>>> in > > > >>> >>> > > > > > >>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> checkpoint. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> While > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> you can infer that TTL is enabled > by > > > >>> reading > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>>> serializer, > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> checkpoint > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> itself only stores the last access > > > time > > > >>> for > > > >>> >>> > > > > > >>>> each > > > >>> >>> > > > > > >>>>>>> value. > > > >>> >>> > > > > > >>>>>>>>> So > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> only > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> thing > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> we can show is the last access > time > > > for > > > >>> each > > > >>> >>> > > > > > >>>>> value. > > > >>> >>> > > > > > >>>>>>> But > > > >>> >>> > > > > > >>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> required > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> for all state backends to store > > this, > > > as > > > >>> >>> they > > > >>> >>> > > > > > >>>> may > > > >>> >>> > > > > > >>>>>>>>> directly > > > >>> >>> > > > > > >>>>>>>>>>>> store > > > >>> >>> > > > > > >>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> expired time. This will also > > increase > > > >>> the > > > >>> >>> > > > > > >>>>>>> difficulty of > > > >>> >>> > > > > > >>>>>>>>>>>>>>> implementation > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> & > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> maintenance. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> This once again reiterates the > > > >>> importance of > > > >>> >>> > > > > > >>>>> unified > > > >>> >>> > > > > > >>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> checkpoints. I’m planning on > adding > > > >>> this, > > > >>> >>> and > > > >>> >>> > > > > > >>>> we > > > >>> >>> > > > > > >>>>> may > > > >>> >>> > > > > > >>>>>>>>>>>> collaborate > > > >>> >>> > > > > > >>>>>>>>>>>>> on > > > >>> >>> > > > > > >>>>>>>>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the future. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata > > Column > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> I'm not in favor of adding a new > > > >>> connector > > > >>> >>> > > > > > >> for > > > >>> >>> > > > > > >>>>>>>> metadata. > > > >>> >>> > > > > > >>>>>>>>>> The > > > >>> >>> > > > > > >>>>>>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> more like one-time information > > instead > > > >>> of a > > > >>> >>> > > > > > >>>>>>> streaming > > > >>> >>> > > > > > >>>>>>>>> data > > > >>> >>> > > > > > >>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>> changes > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> all > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the time, so a single connector > > seems > > > >>> to be > > > >>> >>> > > > > > >> an > > > >>> >>> > > > > > >>>>>>>> overkill. > > > >>> >>> > > > > > >>>>>>>>> It > > > >>> >>> > > > > > >>>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>>>>> easy > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> withdraw a connector if we have a > > > better > > > >>> >>> > > > > > >>>> solution > > > >>> >>> > > > > > >>>>> in > > > >>> >>> > > > > > >>>>>>>>>> future. > > > >>> >>> > > > > > >>>>>>>>>>>> I'm > > > >>> >>> > > > > > >>>>>>>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> familiar with current Catalog > > > >>> capabilities, > > > >>> >>> > > > > > >>>> and if > > > >>> >>> > > > > > >>>>>>> it > > > >>> >>> > > > > > >>>>>>>>> could > > > >>> >>> > > > > > >>>>>>>>>>>>> extract > > > >>> >>> > > > > > >>>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> show some operator-level > information > > > >>> from > > > >>> >>> > > > > > >>>>> savepoint, > > > >>> >>> > > > > > >>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> great. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> If the Catalog can't do that, I > > would > > > >>> >>> > > > > > >> consider > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>> current > > > >>> >>> > > > > > >>>>>>>>>>> FLIP > > > >>> >>> > > > > > >>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>> be a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> compromise solution. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> And if we have that unified > metadata > > > for > > > >>> >>> > > > > > >>>>>>>>>> checkpoint/savepoint > > > >>> >>> > > > > > >>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> future, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> may directly register savepoint in > > > >>> catalog, > > > >>> >>> > > > > > >> and > > > >>> >>> > > > > > >>>>>>> create > > > >>> >>> > > > > > >>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>> source > > > >>> >>> > > > > > >>>>>>>>>>>>>>> without > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> specifying complex columns, as > well > > as > > > >>> >>> > > > > > >> describe > > > >>> >>> > > > > > >>>>> the > > > >>> >>> > > > > > >>>>>>>>>> savepoint > > > >>> >>> > > > > > >>>>>>>>>>>>>> catalog > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> get the metadata. That's a good > > > >>> solution in > > > >>> >>> > > > > > >> my > > > >>> >>> > > > > > >>>>> mind. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> Zakelly > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 10:35 AM > > > >>> Shengkai > > > >>> >>> > > > > > >> Fang > > > >>> >>> > > > > > >>>> < > > > >>> >>> > > > > > >>>>>>>>>>>>> fskm...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Hi Gabor, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > > > >>> >>> > > > > > >>>>>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> I would argue against > introducing a > > > new > > > >>> >>> > > > > > >>>>> connector > > > >>> >>> > > > > > >>>>>>>> type > > > >>> >>> > > > > > >>>>>>>>>>> named > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> savepoint-metadata, as the > existing > > > >>> Catalog > > > >>> >>> > > > > > >>>>>>> mechanism > > > >>> >>> > > > > > >>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>> inherently > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> provide the necessary connector > > > factory > > > >>> >>> > > > > > >>>>>>> capabilities. > > > >>> >>> > > > > > >>>>>>>>>> I’ve > > > >>> >>> > > > > > >>>>>>>>>>>>>> detailed > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> proposal in branch[1]. Please > take > > a > > > >>> moment > > > >>> >>> > > > > > >>>> to > > > >>> >>> > > > > > >>>>>>> review > > > >>> >>> > > > > > >>>>>>>>> it. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> If we introduce a connector named > > > >>> >>> > > > > > >>>>>>>> `savepoint-metadata`, > > > >>> >>> > > > > > >>>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>>> means > > > >>> >>> > > > > > >>>>>>>>>>>>>>> user > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> create a temporary table with > > > connector > > > >>> >>> > > > > > >>>>>>>>>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> connector needs to check whether > > > table > > > >>> >>> > > > > > >>>> schema is > > > >>> >>> > > > > > >>>>>>> same > > > >>> >>> > > > > > >>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> schema > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> proposed in the FLIP. On the > other > > > >>> hand, > > > >>> >>> > > > > > >> it's > > > >>> >>> > > > > > >>>>> not > > > >>> >>> > > > > > >>>>>>>> easy > > > >>> >>> > > > > > >>>>>>>>>> work > > > >>> >>> > > > > > >>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> others > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> users a metadata table with same > > > >>> schema. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://github.com/apache/flink/compare/master...fsk119:flink:state-metadata?expand=1#diff-712a7bc92fe46c405fb0e61b475bb2a005cb7a72bab7df28bbb92744bcb5f465R63 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> Gabor Somogyi < > > > >>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>> 于2025年3月11日周二 > > > >>> >>> > > > > > >>>>>>>>>>>>> 16:56写道： > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Shengkai, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> From directional perspective I > > agree > > > >>> your > > > >>> >>> > > > > > >>>> idea > > > >>> >>> > > > > > >>>>>>> how > > > >>> >>> > > > > > >>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> implemented. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Previously I've mentioned that > TTL > > > >>> >>> > > > > > >>>> information > > > >>> >>> > > > > > >>>>>>> is > > > >>> >>> > > > > > >>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>> exposed > > > >>> >>> > > > > > >>>>>>>>>>>>>> on > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> processor API (which the SQL > state > > > >>> >>> > > > > > >>>> connector > > > >>> >>> > > > > > >>>>>>> uses > > > >>> >>> > > > > > >>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>> read > > > >>> >>> > > > > > >>>>>>>>>>>>> data) > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> and unless somebody show me the > > > >>> opposite > > > >>> >>> > > > > > >>>> this > > > >>> >>> > > > > > >>>>>>> FLIP > > > >>> >>> > > > > > >>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>>> going > > > >>> >>> > > > > > >>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> address > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> this to avoid feature creep. Our > > > users > > > >>> >>> > > > > > >> are > > > >>> >>> > > > > > >>>>> also > > > >>> >>> > > > > > >>>>>>>>>>> interested > > > >>> >>> > > > > > >>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>> TTL > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> so > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> sooner or later we're going to > > > expose > > > >>> it, > > > >>> >>> > > > > > >>>> this > > > >>> >>> > > > > > >>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>> matter > > > >>> >>> > > > > > >>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> scheduling. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > > > >>> >>> > > > > > >>>>>>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Not sure I understand your point > > at > > > >>> all > > > >>> >>> > > > > > >>>>> related > > > >>> >>> > > > > > >>>>>>>>>>>> StateCatalog. > > > >>> >>> > > > > > >>>>>>>>>>>>>>> First > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> all > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I can't agree more that > > StateCatalog > > > >>> is > > > >>> >>> > > > > > >>>> needed > > > >>> >>> > > > > > >>>>>>> and > > > >>> >>> > > > > > >>>>>>>>> is a > > > >>> >>> > > > > > >>>>>>>>>>>>> planned > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> building > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> block in an upcoming > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> FLIP but not sure how can it > help > > > >>> now? No > > > >>> >>> > > > > > >>>>> matter > > > >>> >>> > > > > > >>>>>>>>> what, > > > >>> >>> > > > > > >>>>>>>>>>> your > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> knowledge > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> essential when we add > > StateCatalog. > > > >>> Let > > > >>> >>> > > > > > >> me > > > >>> >>> > > > > > >>>>>>> expose > > > >>> >>> > > > > > >>>>>>>> my > > > >>> >>> > > > > > >>>>>>>>>>>>>>> understanding > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> area: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * First we need create table > > > >>> statements > > > >>> >>> > > > > > >> to > > > >>> >>> > > > > > >>>>>>> access > > > >>> >>> > > > > > >>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>> data > > > >>> >>> > > > > > >>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * When we have that then we can > > add > > > >>> >>> > > > > > >>>>> StateCatalog > > > >>> >>> > > > > > >>>>>>>>> which > > > >>> >>> > > > > > >>>>>>>>>>>> could > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> potentially > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> ease the life of users by for > ex. > > > >>> giving > > > >>> >>> > > > > > >>>>>>>>> off-the-shelf > > > >>> >>> > > > > > >>>>>>>>>>>> tables > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> without > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> sweating with create table > > > statements > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> User expectations: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See state data (this is > > fulfilled > > > >>> with > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>> existing > > > >>> >>> > > > > > >>>>>>>>>>>>>> connector) > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See metadata about state data > > like > > > >>> TTL > > > >>> >>> > > > > > >>>> (this > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>> added > > > >>> >>> > > > > > >>>>>>>>>>>>> as > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> column as you suggested since it > > > >>> belongs > > > >>> >>> > > > > > >> to > > > >>> >>> > > > > > >>>>> the > > > >>> >>> > > > > > >>>>>>>> data) > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> * See metadata about operators > > (this > > > >>> can > > > >>> >>> > > > > > >> be > > > >>> >>> > > > > > >>>>>>> added > > > >>> >>> > > > > > >>>>>>>>> from > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> savepoint-metadata) > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> Important to highlight that > state > > > data > > > >>> >>> > > > > > >>>> table > > > >>> >>> > > > > > >>>>>>> format > > > >>> >>> > > > > > >>>>>>>>>>> differs > > > >>> >>> > > > > > >>>>>>>>>>>>>> from > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> metadata table format. Namely > one > > > >>> table > > > >>> >>> > > > > > >> has > > > >>> >>> > > > > > >>>>> rows > > > >>> >>> > > > > > >>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>>> values > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> another has rows for operators, > > > right? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I think that's the reason why > > you've > > > >>> >>> > > > > > >>>>> pinpointed > > > >>> >>> > > > > > >>>>>>> out > > > >>> >>> > > > > > >>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> suggested > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> metadata columns are somewhat > > > clunky. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> As a conclusion I agree to add > > > >>> >>> > > > > > >>>>> ${state-name}_ttl > > > >>> >>> > > > > > >>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>> column > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> later > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> on > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> since it belongs to the state > > value > > > >>> and > > > >>> >>> > > > > > >>>>> adding a > > > >>> >>> > > > > > >>>>>>>> new > > > >>> >>> > > > > > >>>>>>>>>>> table > > > >>> >>> > > > > > >>>>>>>>>>>>> type > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> (like > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> you > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> suggested similar to PG [1]) > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> for metadata. Please see how > Spark > > > >>> does > > > >>> >>> > > > > > >>>> that > > > >>> >>> > > > > > >>>>> too > > > >>> >>> > > > > > >>>>>>>> [2]. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> If you have better approach then > > > >>> please > > > >>> >>> > > > > > >>>>>>> elaborate > > > >>> >>> > > > > > >>>>>>>>> with > > > >>> >>> > > > > > >>>>>>>>>>> more > > > >>> >>> > > > > > >>>>>>>>>>>>>>> details > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> help me to understand your > point. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in > > TB > > > >>> >>> > > > > > >>>>> savepoints > > > >>> >>> > > > > > >>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>> number > > > >>> >>> > > > > > >>>>>>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> keys > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> be extremely huge but not the > per > > > key > > > >>> >>> > > > > > >>>> state > > > >>> >>> > > > > > >>>>>>>> itself. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> But again, this is a good > feature > > > >>> as-is > > > >>> >>> > > > > > >>>> and > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>> handled > > > >>> >>> > > > > > >>>>>>>>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> separate > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> jira. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> I've just created > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >> https://issues.apache.org/jira/browse/FLINK-37456 > . > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> https://www.postgresql.org/docs/current/view-pg-tables.html > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [2] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Mar 11, 2025 at 3:55 AM > > > >>> Shengkai > > > >>> >>> > > > > > >>>> Fang > > > >>> >>> > > > > > >>>>> < > > > >>> >>> > > > > > >>>>>>>>>>>>>> fskm...@gmail.com > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your > > > response. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Thank you for addressing the > > > >>> >>> > > > > > >> limitations > > > >>> >>> > > > > > >>>>> here. > > > >>> >>> > > > > > >>>>>>>>>>> However, I > > > >>> >>> > > > > > >>>>>>>>>>>>>>> believe > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> be beneficial to further > clarify > > > the > > > >>> >>> > > > > > >> API > > > >>> >>> > > > > > >>>> in > > > >>> >>> > > > > > >>>>>>> this > > > >>> >>> > > > > > >>>>>>>>> FLIP > > > >>> >>> > > > > > >>>>>>>>>>>>>> regarding > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> how > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> users > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> can specify the TTL column. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> One potential approach that > comes > > > to > > > >>> >>> > > > > > >>>> mind is > > > >>> >>> > > > > > >>>>>>>> using > > > >>> >>> > > > > > >>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>>>>>> standardized > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> naming > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> convention such as > > > ${state-name}_ttl > > > >>> >>> > > > > > >> for > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>> column > > > >>> >>> > > > > > >>>>>>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> defines > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> the TTL value. In terms of > > > >>> >>> > > > > > >>>> implementation, > > > >>> >>> > > > > > >>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> listReadableMetadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> function could: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. Read the table’s columns and > > > >>> >>> > > > > > >>>>> configuration, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Extract all defined state > > names, > > > >>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 3. Return a structured list of > > > >>> metadata > > > >>> >>> > > > > > >>>>>>> entries > > > >>> >>> > > > > > >>>>>>>>>>> formatted > > > >>> >>> > > > > > >>>>>>>>>>>>> as > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> ${state-name}_ttl. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> WDYT? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > > > >>> >>> > > > > > >>>>>>>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Introducing a new connector > type > > at > > > >>> >>> > > > > > >> this > > > >>> >>> > > > > > >>>>> stage > > > >>> >>> > > > > > >>>>>>>> may > > > >>> >>> > > > > > >>>>>>>>>>>>>>> unnecessarily > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> complicate > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> the system. Given that every > > table > > > >>> >>> > > > > > >>>> already > > > >>> >>> > > > > > >>>>>>>> belongs > > > >>> >>> > > > > > >>>>>>>>>> to a > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Catalog, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> which > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> designed to provide a Factory > for > > > >>> >>> > > > > > >>>> building > > > >>> >>> > > > > > >>>>>>> source > > > >>> >>> > > > > > >>>>>>>>> or > > > >>> >>> > > > > > >>>>>>>>>>> sink > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> connectors, I > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> propose integrating a dedicated > > > >>> >>> > > > > > >>>> StateCatalog > > > >>> >>> > > > > > >>>>>>>>> instead. > > > >>> >>> > > > > > >>>>>>>>>>>> This > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> approach > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> allow us to: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 1. Leverage the Catalog’s > > existing > > > >>> >>> > > > > > >>>>>>> capabilities > > > >>> >>> > > > > > >>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>> manage > > > >>> >>> > > > > > >>>>>>>>>>>>> TTL > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> (e.g., state names and TTL > logic) > > > >>> >>> > > > > > >> without > > > >>> >>> > > > > > >>>>>>>>> duplicating > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> functionality. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> 2. Provide a unified interface > > for > > > >>> >>> > > > > > >>>> connector > > > >>> >>> > > > > > >>>>>>>>>>>> instantiation > > > >>> >>> > > > > > >>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> handling through the Catalog’s > > > >>> Factory > > > >>> >>> > > > > > >>>>>>> pattern. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Would this design decision > better > > > >>> align > > > >>> >>> > > > > > >>>> with > > > >>> >>> > > > > > >>>>>>> our > > > >>> >>> > > > > > >>>>>>>>>>>>>> architecture’s > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> extensibility and reduce > > > redundancy? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even > in > > TB > > > >>> >>> > > > > > >>>>>>> savepoints > > > >>> >>> > > > > > >>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> number > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> keys > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the > > per > > > >>> key > > > >>> >>> > > > > > >>>>> state > > > >>> >>> > > > > > >>>>>>>>> itself. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good > > feature > > > >>> >>> > > > > > >> as-is > > > >>> >>> > > > > > >>>>> and > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>> handled > > > >>> >>> > > > > > >>>>>>>>>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> separate > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> jira. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> +1 for a separate jira. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < > > > >>> >>> > > > > > >> gabor.g.somo...@gmail.com > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>>>>>>>>> 于2025年3月10日周一 > > > >>> >>> > > > > > >>>>>>>>>>>>>>> 19:05写道： > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Shengkai, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Please see my comments inline. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 3, 2025 at 7:07 AM > > > >>> >>> > > > > > >> Shengkai > > > >>> >>> > > > > > >>>>>>> Fang < > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> fskm...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your > the > > > >>> >>> > > > > > >> FLIP. > > > >>> >>> > > > > > >>>> I > > > >>> >>> > > > > > >>>>>>> have > > > >>> >>> > > > > > >>>>>>>>> some > > > >>> >>> > > > > > >>>>>>>>>>>>>> questions > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> about > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> FLIP: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value > Columns > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> How can users retrieve the > > state > > > >>> >>> > > > > > >> TTL > > > >>> >>> > > > > > >>>>>>>>>> (Time-to-Live) > > > >>> >>> > > > > > >>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>> each > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> value > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> column? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> From my understanding of the > > > >>> >>> > > > > > >> current > > > >>> >>> > > > > > >>>>>>> design, > > > >>> >>> > > > > > >>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>> seems > > > >>> >>> > > > > > >>>>>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> functionality is not > supported. > > > >>> >>> > > > > > >> Could > > > >>> >>> > > > > > >>>>> you > > > >>> >>> > > > > > >>>>>>>>> clarify > > > >>> >>> > > > > > >>>>>>>>>>> if > > > >>> >>> > > > > > >>>>>>>>>>>>>> there > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> are > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> plans > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> address this limitation? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Since the state processor API > is > > > not > > > >>> >>> > > > > > >>>> yet > > > >>> >>> > > > > > >>>>>>>> exposing > > > >>> >>> > > > > > >>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> information > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> would require several steps. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> First, the state processor API > > > >>> >>> > > > > > >> support > > > >>> >>> > > > > > >>>>>>> needs to > > > >>> >>> > > > > > >>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>> added > > > >>> >>> > > > > > >>>>>>>>>>>>>>> which > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> then > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> exposed on the SQL API. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This is definitely a future > > > >>> >>> > > > > > >> improvement > > > >>> >>> > > > > > >>>>>>> which > > > >>> >>> > > > > > >>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>> useful > > > >>> >>> > > > > > >>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> handled > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> in a separate jira. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 2. Metadata Table vs. > Metadata > > > >>> >>> > > > > > >> Column > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> The metadata information > > > described > > > >>> >>> > > > > > >> in > > > >>> >>> > > > > > >>>>> the > > > >>> >>> > > > > > >>>>>>>> FLIP > > > >>> >>> > > > > > >>>>>>>>>>>> appears > > > >>> >>> > > > > > >>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> intended > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> describe the state files > stored > > > at > > > >>> >>> > > > > > >> a > > > >>> >>> > > > > > >>>>>>> specific > > > >>> >>> > > > > > >>>>>>>>>>>> location. > > > >>> >>> > > > > > >>>>>>>>>>>>>> To > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> me, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> concept > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> aligns more closely with > system > > > >>> >>> > > > > > >>>> tables > > > >>> >>> > > > > > >>>>>>> like > > > >>> >>> > > > > > >>>>>>>>>>> pg_tables > > > >>> >>> > > > > > >>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> PostgreSQL > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> or > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the INFORMATION_SCHEMA in > MySQL > > > >>> >>> > > > > > >> [2]. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Adding a new connector with > > > >>> >>> > > > > > >>>>>>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>> is a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> possibility > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> where > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> can create such functionality. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> I'm not against that, just > want > > to > > > >>> >>> > > > > > >>>> have a > > > >>> >>> > > > > > >>>>>>>> common > > > >>> >>> > > > > > >>>>>>>>>>>>> agreement > > > >>> >>> > > > > > >>>>>>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> we > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> like to move that direction. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> (As a side note not just PG > but > > > >>> Spark > > > >>> >>> > > > > > >>>> also > > > >>> >>> > > > > > >>>>>>> has > > > >>> >>> > > > > > >>>>>>>>>>> similar > > > >>> >>> > > > > > >>>>>>>>>>>>>>> approach > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> and I > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> basically like the idea). > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> If we would go that direction > > > >>> >>> > > > > > >> savepoint > > > >>> >>> > > > > > >>>>>>>> metadata > > > >>> >>> > > > > > >>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>> reached > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> way > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> that one row would represent > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> an operator with it's values > > > >>> >>> > > > > > >> something > > > >>> >>> > > > > > >>>>> like > > > >>> >>> > > > > > >>>>>>>> this: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┐ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > │operatorN│operatorU│operatorH│paralleli│maxParall│subtaskSt│coordinat│totalSta│ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ame │id │ash > > │sm > > > >>> >>> > > > > > >>>>>>> │elism > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │atesCount│orStateSi│tesSizeI│ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │ > │ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │zeInBytes│nBytes │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │Source: > │datagen-s│47aee9439│2 > > > >>> >>> > > > > > >>>>> │128 > > > >>> >>> > > > > > >>>>>>>>>> │2 > > > >>> >>> > > > > > >>>>>>>>>>>>>>> │16 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │546 │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > │datagen-s│ource-uid│4d6ea26e2│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ource │ > │d544bef0a│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │37bb5 > │ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > │long-udf-│long-udf-│6ed3f40bf│2 > > > >>> >>> > > > > > >>>>> │128 > > > >>> >>> > > > > > >>>>>>>>>> │2 > > > >>> >>> > > > > > >>>>>>>>>>>>>>> │0 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> │0 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > │with-mast│with-mast│f3c8dfcdf│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │er-hook > │er-hook-u│cb95128a1│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │id │018f1 > │ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > │value-pro│value-pro│ca4f5fe9a│2 > > > >>> >>> > > > > > >>>>> │128 > > > >>> >>> > > > > > >>>>>>>>>> │2 > > > >>> >>> > > > > > >>>>>>>>>>>>>>> │0 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │40726 │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │cess │cess-uid > │637b656f0│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ > │9ea78b3e7│ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ │ │a15b9 > │ > > > >>> >>> > > > > > >>>> │ > > > >>> >>> > > > > > >>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> │ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This table can then be joined > > with > > > >>> >>> > > > > > >> the > > > >>> >>> > > > > > >>>>>>> actually > > > >>> >>> > > > > > >>>>>>>>>>>> existing > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> `savepoint` > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> connector created tables based > > on > > > >>> UID > > > >>> >>> > > > > > >>>> hash > > > >>> >>> > > > > > >>>>>>>> (which > > > >>> >>> > > > > > >>>>>>>>>> is > > > >>> >>> > > > > > >>>>>>>>>>>>> unique > > > >>> >>> > > > > > >>>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> always > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> exists). > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> This would mean that the > already > > > >>> >>> > > > > > >>>> existing > > > >>> >>> > > > > > >>>>>>> table > > > >>> >>> > > > > > >>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>> need > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> only a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> single > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> metadata column which is the > UID > > > >>> >>> > > > > > >> hash. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> WDYT? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> @zakelly, plz share your > > thoughts > > > >>> >>> > > > > > >> too. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> If we opt to use metadata > > > columns, > > > >>> >>> > > > > > >>>> every > > > >>> >>> > > > > > >>>>>>>> record > > > >>> >>> > > > > > >>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>> table > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> would > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> end > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> up > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> having identical values for > > these > > > >>> >>> > > > > > >>>>> columns > > > >>> >>> > > > > > >>>>>>>>> (please > > > >>> >>> > > > > > >>>>>>>>>>>>> correct > > > >>> >>> > > > > > >>>>>>>>>>>>>>> me > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> if > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> I’m > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> mistaken). On the other hand, > > the > > > >>> >>> > > > > > >>>> state > > > >>> >>> > > > > > >>>>>>>>> connector > > > >>> >>> > > > > > >>>>>>>>>>>>>> requires > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> users > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> specify > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> an operator UID or operator > UID > > > >>> >>> > > > > > >> hash, > > > >>> >>> > > > > > >>>>>>> after > > > >>> >>> > > > > > >>>>>>>>> which > > > >>> >>> > > > > > >>>>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>>>>> outputs > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> user-defined > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> values in its records. This > > > >>> >>> > > > > > >> approach > > > >>> >>> > > > > > >>>>> feels > > > >>> >>> > > > > > >>>>>>>>>> somewhat > > > >>> >>> > > > > > >>>>>>>>>>>>>>> redundant > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> me. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> If we would add a new > > > >>> >>> > > > > > >>>> `savepoint-metadata` > > > >>> >>> > > > > > >>>>>>>>>> connector > > > >>> >>> > > > > > >>>>>>>>>>>> then > > > >>> >>> > > > > > >>>>>>>>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> addressed. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> On the other hand UID and UID > > hash > > > >>> >>> > > > > > >> are > > > >>> >>> > > > > > >>>>>>> having > > > >>> >>> > > > > > >>>>>>>>>>> either-or > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> relationship > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> from > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> config perspective, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> so when a user provides the > UID > > > then > > > >>> >>> > > > > > >>>>> he/she > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>>> interested > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> hash > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> for further calculations > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> (the whole Flink internals are > > > >>> >>> > > > > > >>>> depending > > > >>> >>> > > > > > >>>>> on > > > >>> >>> > > > > > >>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>> hash). > > > >>> >>> > > > > > >>>>>>>>>>>>>>> Printing > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> out > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> human readable UID > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> is an explicit requirement > from > > > the > > > >>> >>> > > > > > >>>> user > > > >>> >>> > > > > > >>>>>>> side > > > >>> >>> > > > > > >>>>>>>>>> because > > > >>> >>> > > > > > >>>>>>>>>>>>>> hashes > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> are > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> not > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> human > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> readable. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 3. Handling LIST and MAP > States > > > in > > > >>> >>> > > > > > >>>> the > > > >>> >>> > > > > > >>>>>>> State > > > >>> >>> > > > > > >>>>>>>>>>>> Connector > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I have concerns about how the > > > >>> >>> > > > > > >> current > > > >>> >>> > > > > > >>>>>>> design > > > >>> >>> > > > > > >>>>>>>>>>> handles > > > >>> >>> > > > > > >>>>>>>>>>>>> LIST > > > >>> >>> > > > > > >>>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> MAP > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> states. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Specifically, the state > > connector > > > >>> >>> > > > > > >>>> uses > > > >>> >>> > > > > > >>>>>>> Flink > > > >>> >>> > > > > > >>>>>>>>>> SQL’s > > > >>> >>> > > > > > >>>>>>>>>>>> MAP > > > >>> >>> > > > > > >>>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> ARRAY > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> types, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> which implies that it > attempts > > to > > > >>> >>> > > > > > >>>> load > > > >>> >>> > > > > > >>>>>>> entire > > > >>> >>> > > > > > >>>>>>>>> MAP > > > >>> >>> > > > > > >>>>>>>>>>> or > > > >>> >>> > > > > > >>>>>>>>>>>>> LIST > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> states > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> into > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> memory. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> However, in many real-world > > > >>> >>> > > > > > >>>> scenarios, > > > >>> >>> > > > > > >>>>>>> these > > > >>> >>> > > > > > >>>>>>>>>> states > > > >>> >>> > > > > > >>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>> grow > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> very > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> large. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Typically, the state API > > > addresses > > > >>> >>> > > > > > >>>> this > > > >>> >>> > > > > > >>>>> by > > > >>> >>> > > > > > >>>>>>>>>>> providing > > > >>> >>> > > > > > >>>>>>>>>>>> an > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> iterator > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> traverse elements within the > > > state > > > >>> >>> > > > > > >>>>>>>>> incrementally. > > > >>> >>> > > > > > >>>>>>>>>>> I’m > > > >>> >>> > > > > > >>>>>>>>>>>>>>> unsure > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> whether > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> I’ve > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> missed something in FLIP-496 > or > > > >>> >>> > > > > > >>>>> FLIP-512, > > > >>> >>> > > > > > >>>>>>> but > > > >>> >>> > > > > > >>>>>>>>> it > > > >>> >>> > > > > > >>>>>>>>>>>> seems > > > >>> >>> > > > > > >>>>>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> current > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> design might struggle with > > > >>> >>> > > > > > >>>> scalability > > > >>> >>> > > > > > >>>>> in > > > >>> >>> > > > > > >>>>>>>> such > > > >>> >>> > > > > > >>>>>>>>>>> cases. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> You see it good, the current > > > >>> >>> > > > > > >>>>> implementation > > > >>> >>> > > > > > >>>>>>>> keeps > > > >>> >>> > > > > > >>>>>>>>>>> state > > > >>> >>> > > > > > >>>>>>>>>>>>>> for a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> single > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> key > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> in > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> memory. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Back in the days we've > > considered > > > >>> >>> > > > > > >> this > > > >>> >>> > > > > > >>>>>>>> potential > > > >>> >>> > > > > > >>>>>>>>>>> issue > > > >>> >>> > > > > > >>>>>>>>>>>>> and > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> concluded > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> this is not necessarily > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> needed for the initial version > > and > > > >>> >>> > > > > > >> can > > > >>> >>> > > > > > >>>> be > > > >>> >>> > > > > > >>>>>>> done > > > >>> >>> > > > > > >>>>>>>>> as a > > > >>> >>> > > > > > >>>>>>>>>>>> later > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> improvement. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even > in > > TB > > > >>> >>> > > > > > >>>>>>> savepoints > > > >>> >>> > > > > > >>>>>>>>> that > > > >>> >>> > > > > > >>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>> number > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> of > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> keys > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> can > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the > > per > > > >>> key > > > >>> >>> > > > > > >>>>> state > > > >>> >>> > > > > > >>>>>>>>> itself. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good > > feature > > > >>> >>> > > > > > >> as-is > > > >>> >>> > > > > > >>>>> and > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>>>> handled > > > >>> >>> > > > > > >>>>>>>>>>>>>>> in a > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> separate > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> jira. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Shengkai > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > https://www.postgresql.org/docs/current/view-pg-tables.html > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> [2] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < > > > >>> >>> > > > > > >>>>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>> 于2025年3月3日周一 > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> 02:00写道： > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Zakelly, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> In order to shoot for > > simplicity > > > >>> >>> > > > > > >>>>>>> `METADATA > > > >>> >>> > > > > > >>>>>>>>>>> VIRTUAL` > > > >>> >>> > > > > > >>>>>>>>>>>>> as > > > >>> >>> > > > > > >>>>>>>>>>>>>>> key > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> words > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> definition is the target. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> When it's not super complex > > the > > > >>> >>> > > > > > >>>> latter > > > >>> >>> > > > > > >>>>>>> can > > > >>> >>> > > > > > >>>>>>>> be > > > >>> >>> > > > > > >>>>>>>>>>> added > > > >>> >>> > > > > > >>>>>>>>>>>>>> too. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 2, 2025 at > 3:37 PM > > > >>> >>> > > > > > >>>> Zakelly > > > >>> >>> > > > > > >>>>>>> Lan > > > >>> >>> > > > > > >>>>>>>> < > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> zakelly....@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Gabor, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> +1 for this. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Will the metadata column > use > > > >>> >>> > > > > > >>>>> `METADATA > > > >>> >>> > > > > > >>>>>>>>>> VIRTUAL` > > > >>> >>> > > > > > >>>>>>>>>>>> as > > > >>> >>> > > > > > >>>>>>>>>>>>>> key > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> words > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> definition, or `METADATA > FROM > > > >>> >>> > > > > > >> xxx > > > >>> >>> > > > > > >>>>>>>> VIRTUAL` > > > >>> >>> > > > > > >>>>>>>>>> for > > > >>> >>> > > > > > >>>>>>>>>>>>>>> renaming, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> just > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> like > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> the > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Kafka table? > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Zakelly > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Mar 1, 2025 at > > 1:31 PM > > > >>> >>> > > > > > >>>> Gabor > > > >>> >>> > > > > > >>>>>>>>> Somogyi > > > >>> >>> > > > > > >>>>>>>>>> < > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> gabor.g.somo...@gmail.com> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start a > > > >>> >>> > > > > > >> discussion > > > >>> >>> > > > > > >>>> of > > > >>> >>> > > > > > >>>>>>>>> FLIP-512: > > > >>> >>> > > > > > >>>>>>>>>>> Add > > > >>> >>> > > > > > >>>>>>>>>>>>>> meta > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> information > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> to > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> SQL > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> state connector [1]. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to add your > > > >>> >>> > > > > > >> thoughts > > > >>> >>> > > > > > >>>> to > > > >>> >>> > > > > > >>>>>>> make > > > >>> >>> > > > > > >>>>>>>>> this > > > >>> >>> > > > > > >>>>>>>>>>>>> feature > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> better. > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> BR, > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> G > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>>> > > > >>> >>> > > > > > >>>>>>>>> > > > >>> >>> > > > > > >>>>>>>> > > > >>> >>> > > > > > >>>>>>> > > > >>> >>> > > > > > >>>>>> > > > >>> >>> > > > > > >>>>> > > > >>> >>> > > > > > >>>> > > > >>> >>> > > > > > >>> > > > >>> >>> > > > > > >> > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > > >>> >>> > > > > > > > >>> >>> > > > > > > >>> >>> > > > > > >>> >>> > > > > >>> >>> > > > >>> >> > > > >>> > > > >> > > > > > >
