Many thanks for your reminder, Leonard. Here's the link I mentioned[1]. Best, Shengkai
[1] https://github.com/apache/flink/pull/26358 Leonard Xu <xbjt...@gmail.com> 于2025年3月27日周四 10:05写道： > Your link is broken, Shengkai > > Best, > Leonard > > > 2025年3月27日 10:01，Shengkai Fang <fskm...@gmail.com> 写道： > > > > Hi, All. > > > > I write a simple demo to illustrate my idea. Hope this helps. > > > > Best, > > Shengkai > > > > > https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1 > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月26日周三 15:54写道： > > > >>> I'm fine with a seperate SQL connector for metadata, so maybe we could > >> update the FLIP about our discussion? > >> > >> Sorry, I've forgotten this part. Yeah, no matter we choose I'm going to > >> update the FLIP. > >> > >> G > >> > >> > >> On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi < > gabor.g.somo...@gmail.com> > >> wrote: > >> > >>> Hi All, > >>> > >>> I've also lack of the knowledge of PTF so I've read just the motivation > >>> part: > >>> > >>> "The SQL 2016 standard introduced a way of defining custom SQL > operators > >>> defined by ISO/IEC 19075-7:2021 (Part 7: Polymorphic table functions). > >>> ~200 pages define how this new kind of function can consume and produce > >>> tables with various execution properties. > >>> Unfortunately, this part of the standard is not publicly available." > >>> > >>> Of course we can take a look at some examples but do we really want to > >>> expose state data with this construct > >>> which is described in ~200 pages and part of the standard is not > publicly > >>> available? 🙂 > >>> I mean the dataset is couple of rows and the use-case is join with > >> another > >>> table like with state data. > >>> If somebody can give advantages I would buy that but from my limited > >>> understanding this would be an overkill here. > >>> > >>> BR, > >>> G > >>> > >>> > >>> On Wed, Mar 26, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> > wrote: > >>> > >>>> Hi Zakelly , Shengkai! > >>>> > >>>> I don't know too much about PTFs, it would be interesting to see how > the > >>>> usage would look in practice. > >>>> > >>>> Do you have some mockup/example in mind how the PTF would look for > >> example > >>>> when want to: > >>>> - Simply display/aggregate whats in the metadata > >>>> - Join keyed state with some metadata columns > >>>> > >>>> Thanks > >>>> Gyula > >>>> > >>>> On Wed, Mar 26, 2025 at 7:33 AM Zakelly Lan <zakelly....@gmail.com> > >>>> wrote: > >>>> > >>>>> Hi everyone, > >>>>> > >>>>> I'm fine with a seperate SQL connector for metadata, so maybe we > could > >>>>> update the FLIP about our discussion? And Shengkai provides a PTF > >>>>> implementation, does that also meet the requirement? > >>>>> > >>>>> > >>>>> Best, > >>>>> Zakelly > >>>>> > >>>>> On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi < > >>>> gabor.g.somo...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> @Zakelly: Gyula summarised it correctly what I meant so please treat > >>>> the > >>>>>> content as mine. > >>>>>> As an addition I'm not against to add CLI at all, I'm just stating > >>>> that > >>>>> in > >>>>>> some cases like this, users would like to have > >>>>>> a self-serving solution where they can provide SQL statements which > >>>> can > >>>>>> trigger alerts automatically. > >>>>>> > >>>>>> My personal opinion is that CLI would be beneficial for several > >>>> cases. A > >>>>>> good example is when users want to restart job > >>>>>> from specific Kafka offsets which are persisted in a savepoint. For > >>>> such > >>>>>> scenario users are more than happy since they > >>>>>> expect manual intervention with full control. So all in all one can > >>>> count > >>>>>> on my +1 when CLI FLIP would come up... > >>>>>> > >>>>>> BR, > >>>>>> G > >>>>>> > >>>>>> > >>>>>> On Thu, Mar 20, 2025 at 8:20 AM Gyula Fóra <gyula.f...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>>> Hi! > >>>>>>> > >>>>>>> @Zakelly Lan <zakelly....@gmail.com> > >>>>>>> I think what Gabor means is that users want to have predefined SQL > >>>>> scripts > >>>>>>> to perform state analysis tasks to debug/identify problems. > >>>>>>> Such as write a SQL script that joins the metadata table with the > >>>> state > >>>>>>> and > >>>>>>> do some analytics on it. > >>>>>>> > >>>>>>> If we have a meta table then the SQL script that can do this is > >> fixed > >>>>> and > >>>>>>> users can trigger this on demand by simply providing a new > >> savepoint > >>>>> path. > >>>>>>> > >>>>>>> If we have a different mechanism to extract metadata that is not > >> SQL > >>>>>>> native > >>>>>>> then manual steps need to be executed and a custom SQL script would > >>>> need > >>>>>>> to > >>>>>>> be written that adds the manually extracted metadata into the > >> script. > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Gyula > >>>>>>> > >>>>>>> On Thu, Mar 20, 2025 at 4:32 AM Zakelly Lan <zakelly....@gmail.com > >>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> Thanks for your answers! Getting everyone aligned on this topic > >> is > >>>>>>>> challenging, but it’s definitely worth the effort since it will > >>>> help > >>>>>>>> streamline things moving forward. > >>>>>>>> > >>>>>>>> @Gabor are you saying that users are using some scripts to define > >>>> the > >>>>>>> SQL > >>>>>>>> metadata connector and get the information, right? If so, would a > >>>> CLI > >>>>>>> tool > >>>>>>>> be more convenient? It's easy to invoke and can get the result > >>>>> swiftly. > >>>>>>> And > >>>>>>>> there should be some other systems to track the checkpoint > >> lineage > >>>> and > >>>>>>>> analyze if there are outliers in metadata (e.g. state size of one > >>>>>>> operator) > >>>>>>>> right? Well, maybe I missed something so please correct me if I'm > >>>>> wrong. > >>>>>>>> > >>>>>>>> I think the overall vision in Flink SQL is to provide a SQL > >> native > >>>>>>>>> environment where we can serve complex use-cases like you would > >>>>> expect > >>>>>>>> in a > >>>>>>>>> regular database. > >>>>>>>> > >>>>>>>> > >>>>>>>> @Gyula Well, this is a good point. From the perspective of > >>>>> comprehensive > >>>>>>>> SQL experience, I'd +1 for treating metadata as data. Although I > >>>> doubt > >>>>>>> if > >>>>>>>> there is a need for processing metadata, I won't be against a > >>>> separate > >>>>>>>> connector. > >>>>>>>> > >>>>>>>> Regarding the CLI tool, I still think it’s worth implementing. > >>>> Such a > >>>>>>> tool > >>>>>>>> could provide savepoint information before resuming from a > >>>> savepoint, > >>>>>>> which > >>>>>>>> would enhance the user experience in CLI-based workflows. It > >> would > >>>> be > >>>>>>> good > >>>>>>>> if someone could implement this feature. We shouldn’t worry about > >>>>>>> whether > >>>>>>>> this tool might be retired in the future. Regardless of the > >>>> SQL-based > >>>>>>>> solution we eventually adopt, this capability will remain > >> essential > >>>>> for > >>>>>>> CLI > >>>>>>>> users. This is another topic. > >>>>>>>> > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Zakelly > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Mar 20, 2025 at 10:37 AM Shengkai Fang < > >> fskm...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi. > >>>>>>>>> > >>>>>>>>> After reading the doc[1], I think Spark provides a function for > >>>>> users > >>>>>>> to > >>>>>>>>> consume the metadata from the savepoint. In Flink SQL, similar > >>>>>>>>> functionality is implemented through Polymorphic Table > >> Functions > >>>>>>> (PTF) as > >>>>>>>>> proposed in FLIP-440[2]. Below is a code example[3] > >> illustrating > >>>>> this > >>>>>>>>> concept: > >>>>>>>>> > >>>>>>>>> ``` > >>>>>>>>> public static class ScalarArgsFunction extends > >>>>>>>>> TestProcessTableFunctionBase { > >>>>>>>>> public void eval(Integer i, Boolean b) { > >>>>>>>>> collectObjects(i, b); > >>>>>>>>> } > >>>>>>>>> } > >>>>>>>>> ``` > >>>>>>>>> > >>>>>>>>> ``` > >>>>>>>>> INSERT INTO sink SELECT * FROM f(i => 42, b => CAST('TRUE' AS > >>>>>>> BOOLEAN)) > >>>>>>>>> `` > >>>>>>>>> > >>>>>>>>> So we can add a builtin function named `read_state_metadata` to > >>>> read > >>>>>>>>> savepoint data. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Shengkai > >>>>>>>>> > >>>>>>>>> [1] > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.databricks.com/aws/en/structured-streaming/read-state?language=SQL > >>>>>>>>> [2] > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=298781093 > >>>>>>>>> [3] > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java#L140 > >>>>>>>>> > >>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月19日周三 18:37写道： > >>>>>>>>> > >>>>>>>>>> Hi All! > >>>>>>>>>> > >>>>>>>>>> Thank you for the answers and concerns from everyone. > >>>>>>>>>> > >>>>>>>>>> On the CLI vs State Metadata Connector/Table question I would > >>>> also > >>>>>>> like > >>>>>>>>> to > >>>>>>>>>> step back a little and look at the bigger picture. > >>>>>>>>>> > >>>>>>>>>> I think the overall vision in Flink SQL is to provide a SQL > >>>> native > >>>>>>>>>> environment where we can serve complex use-cases like you > >> would > >>>>>>> expect > >>>>>>>>> in a > >>>>>>>>>> regular database. > >>>>>>>>>> Most features, developments in the recent years have gone > >> this > >>>>> way. > >>>>>>>>>> > >>>>>>>>>> The State Metadata Table would be a natural and > >> straightforward > >>>>> fit > >>>>>>>> here. > >>>>>>>>>> So from my side, +1 for that. > >>>>>>>>>> > >>>>>>>>>> However I could understand if we are not ready to add a new > >>>>>>>>>> connector/format due to maintenance concerns (and in general > >>>>> concern > >>>>>>>>> about > >>>>>>>>>> the design). > >>>>>>>>>> If that's the issue then we should spend more time on the > >>>> design > >>>>> to > >>>>>>> get > >>>>>>>>>> comfortable with the approach and seek feedback from the > >> wider > >>>>>>>> community > >>>>>>>>>> > >>>>>>>>>> I am -1 for the CLI/tooling approach as that will not provide > >>>> the > >>>>>>>>>> featureset we are looking for that is not already covered by > >>>> the > >>>>>>> Java > >>>>>>>>>> connector. And that approach would come with the same > >>>> maintenance > >>>>>>>>>> implications. > >>>>>>>>>> > >>>>>>>>>> Cheers > >>>>>>>>>> Gyula > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, Mar 19, 2025 at 11:24 AM Gabor Somogyi < > >>>>>>>>> gabor.g.somo...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Zaklely, Shengkai > >>>>>>>>>>> > >>>>>>>>>>> Several topics are going on so adding gist answers to them. > >>>> When > >>>>>>> some > >>>>>>>>>> topic > >>>>>>>>>>> is not touched please highlight it. > >>>>>>>>>>> > >>>>>>>>>>> @Shengkai: I've read through all the previous FLIPs related > >>>>>>> catalogs > >>>>>>>>> and > >>>>>>>>>> if > >>>>>>>>>>> we would like to keep the concepts there > >>>>>>>>>>> then one-to-one mapping relationship between savepoint and > >>>>> catalog > >>>>>>>> is a > >>>>>>>>>>> reasonable direction. In short I'm happy that > >>>>>>>>>>> you've highlighted this and agree as a whole. I've written > >> it > >>>>> down > >>>>>>>>>>> previously, just want to double confirm that state catalog > >> is > >>>>>>>>>>> essential and planned. When we reach this point then your > >>>> input > >>>>> is > >>>>>>>> more > >>>>>>>>>>> than welcome. > >>>>>>>>>>> > >>>>>>>>>>> @Zakelly: We've tried the CLI and separate library > >> approaches > >>>>> with > >>>>>>>>> users > >>>>>>>>>>> already and these are not something which is welcome > >> because > >>>> of > >>>>>>> the > >>>>>>>>>>> following: > >>>>>>>>>>> * Users want to have automated tasks and not manual > >>>> CLI/library > >>>>>>>> output > >>>>>>>>>>> parsing. This can be hacked around but our experience is > >>>>> negative > >>>>>>> on > >>>>>>>>> this > >>>>>>>>>>> because it's just brittle. > >>>>>>>>>>> * From development perspective It's way much bigger effort > >>>> than > >>>>> a > >>>>>>>>>> connector > >>>>>>>>>>> (hard to test, packaging/version handling is and extra > >> layer > >>>> of > >>>>>>>>>> complexity, > >>>>>>>>>>> external FS authentication is pain for users, expecting > >> them > >>>> to > >>>>>>>>> download > >>>>>>>>>>> savepoints also) > >>>>>>>>>>> * Purely personal opinion but if we would find better ways > >>>> later > >>>>>>> then > >>>>>>>>>>> retire a CLI is not more lightweight than retire a > >> connector > >>>>>>>>>>> > >>>>>>>>>>>> It would be great if you give some examples on how user > >>>> could > >>>>>>>>> leverage > >>>>>>>>>>> the separate connector to process the metadata. > >>>>>>>>>>> > >>>>>>>>>>> The most simplest cases: > >>>>>>>>>>> * give me the overgroving state uids > >>>>>>>>>>> * give me the not known (new or renamed) state uids > >>>>>>>>>>> * give me the state uids where state size drastically > >> dropped > >>>>>>> compare > >>>>>>>>> to > >>>>>>>>>> a > >>>>>>>>>>> previous savepoint (accidental state loss) > >>>>>>>>>>> > >>>>>>>>>>> Since it was mentioned: as a general offtopic teaser, yeah > >> it > >>>>>>> would > >>>>>>>> be > >>>>>>>>>> good > >>>>>>>>>>> to have some sort of checkpoint/savepoint lineage or > >> however > >>>> we > >>>>>>> call > >>>>>>>>> it. > >>>>>>>>>>> Since we've not yet reached this point there are no > >> technical > >>>>>>>> details, > >>>>>>>>>> it's > >>>>>>>>>>> more like a vision. It's a common pattern that > >>>>>>>>>>> jobs are physically running but somehow the state > >> processing > >>>> is > >>>>>>> stuck > >>>>>>>>> and > >>>>>>>>>>> it would be good to add some way to find it out > >>>> automatically. > >>>>>>>>>>> The important saying here is automation and not manual > >>>>> evaluation > >>>>>>>> since > >>>>>>>>>>> handling 10k+ jobs is just not allowing that. > >>>>>>>>>>> > >>>>>>>>>>> BR, > >>>>>>>>>>> G > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Mar 19, 2025 at 6:46 AM Shengkai Fang < > >>>>> fskm...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi, All. > >>>>>>>>>>>> > >>>>>>>>>>>> About State Catalog, I want to share more thoughts about > >>>> this. > >>>>>>>>>>>> > >>>>>>>>>>>> In the initial design concept, I understood that a > >>>> savepoint > >>>>>>> and a > >>>>>>>>>> state > >>>>>>>>>>>> catalog have a one-to-one mapping relationship. Each > >>>> operator > >>>>>>>>>> corresponds > >>>>>>>>>>>> to a database, and the state of each operator is > >>>> represented > >>>>> as > >>>>>>>>>>> individual > >>>>>>>>>>>> tables. The rationale behind this design is: > >>>>>>>>>>>> > >>>>>>>>>>>> *State Diversity*: An operator may involve multiple types > >>>> of > >>>>>>>> states. > >>>>>>>>>> For > >>>>>>>>>>>> example, in our VVR design, a "multi-join" operator uses > >>>> keyed > >>>>>>>> states > >>>>>>>>>> for > >>>>>>>>>>>> two input streams and a broadcast state for the third > >>>> stream. > >>>>>>> This > >>>>>>>>>> makes > >>>>>>>>>>> it > >>>>>>>>>>>> challenging to represent all states of an operator > >> within a > >>>>>>> single > >>>>>>>>>> table. > >>>>>>>>>>>> *Scalability*: Internally, an operator might have > >> multiple > >>>>> keyed > >>>>>>>>> states > >>>>>>>>>>>> (e.g., value state and list state). However, large list > >>>> states > >>>>>>> may > >>>>>>>>> not > >>>>>>>>>>> fit > >>>>>>>>>>>> entirely in memory. To address this, we recommend > >>>> implementing > >>>>>>> each > >>>>>>>>>> state > >>>>>>>>>>>> as a separate table. > >>>>>>>>>>>> > >>>>>>>>>>>> To resolve the loosely coupled relationships between > >>>> operator > >>>>>>>> states, > >>>>>>>>>> we > >>>>>>>>>>>> propose embedding predefined views within the catalog. > >>>> These > >>>>>>> views > >>>>>>>>>>> simplify > >>>>>>>>>>>> user understanding of operator implementations and > >> provide > >>>> a > >>>>>>> more > >>>>>>>>>>> intuitive > >>>>>>>>>>>> perspective. For instance, a join operator may have > >>>> multiple > >>>>>>> state > >>>>>>>>>>>> implementations (depending on whether the join key > >> includes > >>>>>>> unique > >>>>>>>>>>>> attributes), but users primarily care about the data > >>>>> associated > >>>>>>>> with > >>>>>>>>> a > >>>>>>>>>>>> specific join key across input streams. > >>>>>>>>>>>> > >>>>>>>>>>>> Returning to the one-to-one mapping between savepoints > >> and > >>>>>>>> catalogs, > >>>>>>>>> we > >>>>>>>>>>> aim > >>>>>>>>>>>> to manage multiple user state catalogs through a catalog > >>>>> store. > >>>>>>>> When > >>>>>>>>> a > >>>>>>>>>>> user > >>>>>>>>>>>> triggers a savepoint for a job on the platform: > >>>>>>>>>>>> > >>>>>>>>>>>> 1. The platform sends a REST request to the JobManager. > >>>>>>>>>>>> 2. Simultaneously, it registers a new state catalog in > >> the > >>>>>>> catalog > >>>>>>>>>> store, > >>>>>>>>>>>> enabling immediate analysis of state data on the > >> platform. > >>>>>>>>>>>> 3. Deleting a savepoint would also trigger the removal of > >>>> its > >>>>>>>>>> associated > >>>>>>>>>>>> catalog. > >>>>>>>>>>>> > >>>>>>>>>>>> This vision assumes that states are self-describing or > >>>> that a > >>>>>>> state > >>>>>>>>>>>> metaservice is introduced to analyze savepoint > >> structures. > >>>>>>>>>>>> > >>>>>>>>>>>>> How can users create logic to identify differences > >>>> between > >>>>>>>> multiple > >>>>>>>>>>>> savepoints? > >>>>>>>>>>>> > >>>>>>>>>>>> Since savepoints and state catalogs are one-to-one > >> mapped, > >>>>> users > >>>>>>>> can > >>>>>>>>>>> query > >>>>>>>>>>>> metadata via their respective catalogs. For example: > >>>>>>>>>>>> > >>>>>>>>>>>> 1. > >>>>> `savepoint-${id}`.`system`.`metadata_table`.`<operator-name>` > >>>>>>>>>> provides > >>>>>>>>>>>> operator-specific metadata (e.g., state size, type). > >>>>>>>>>>>> 2. Comparing metadata tables (e.g., schema versions, > >> state > >>>>> entry > >>>>>>>>>> counts) > >>>>>>>>>>>> across catalogs reveals structural or quantitative > >>>>> differences. > >>>>>>>>>>>> 3. For deeper analysis, users could write SQL queries to > >>>>> compare > >>>>>>>>>> specific > >>>>>>>>>>>> state partitions or leverage the metaservice to track > >> state > >>>>>>>> evolution > >>>>>>>>>>>> (e.g., added/removed operators, modified state > >>>>> configurations). > >>>>>>>>>>>> > >>>>>>>>>>>> If we plan to introduce a state catalog in the future, I > >>>> would > >>>>>>> lean > >>>>>>>>>>> toward > >>>>>>>>>>>> using metadata tables. If a utility tool can address the > >>>>>>> challenges > >>>>>>>>> we > >>>>>>>>>>>> face, could we avoid introducing an additional connector? > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Shengkai > >>>>>>>>>>>> > >>>>>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月17日周一 20:25写道： > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi All! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Without going into too much detail here are my 2 cents > >>>>>>> regarding > >>>>>>>>> the > >>>>>>>>>>>>> virtual column / catalog metadata / table (connector) > >>>>>>> discussion > >>>>>>>>> for > >>>>>>>>>>> the > >>>>>>>>>>>>> State metadata. > >>>>>>>>>>>>> > >>>>>>>>>>>>> State metadata such as the types of states, their > >>>>> properties, > >>>>>>>>> names, > >>>>>>>>>>>> sizes > >>>>>>>>>>>>> etc are all valuable information that can be used to > >>>> enrich > >>>>>>> the > >>>>>>>>>>>>> computations we do on state. > >>>>>>>>>>>>> We can either analyze it standalone (such as discover > >>>>>>> anomalies, > >>>>>>>>> for > >>>>>>>>>>>> large > >>>>>>>>>>>>> jobs with many states), across multiple savepoints > >>>> (discover > >>>>>>> how > >>>>>>>>>> state > >>>>>>>>>>>>> changed over time) or by joining it with keyed or > >>>> non-keyed > >>>>>>> state > >>>>>>>>>> data > >>>>>>>>>>> to > >>>>>>>>>>>>> serve more complex queries on the state. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The only solution that seems to serve all these > >> use-cases > >>>>> and > >>>>>>>>>>>> requirements > >>>>>>>>>>>>> in a straightforward and SQL canonical way is to simply > >>>>> expose > >>>>>>>> the > >>>>>>>>>>> state > >>>>>>>>>>>>> metadata as a separate table. This is a metadata table > >>>> but > >>>>> you > >>>>>>>> can > >>>>>>>>>> also > >>>>>>>>>>>>> think of it as data table, it makes no practical > >>>> difference > >>>>>>> here. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Once we have a catalog later, the catalog can offer > >> this > >>>>> table > >>>>>>>> out > >>>>>>>>> of > >>>>>>>>>>> the > >>>>>>>>>>>>> box, the same way databases provide metadata tables. > >> For > >>>>> this > >>>>>>> to > >>>>>>>>> work > >>>>>>>>>>>>> however we need another, simpler connector that creates > >>>> this > >>>>>>>> table. > >>>>>>>>>>>>> > >>>>>>>>>>>>> +1 for state metadata as a separate connector/table, > >>>> instead > >>>>>>> of > >>>>>>>>>> adding > >>>>>>>>>>>>> virtual columns and adhoc catalog metadata that is hard > >>>> to > >>>>> use > >>>>>>>> in a > >>>>>>>>>>> large > >>>>>>>>>>>>> number of queries. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>> Gyula > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Mar 17, 2025 at 12:44 PM Gabor Somogyi < > >>>>>>>>>>>> gabor.g.somo...@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I’m planning on adding this, and we may collaborate > >>>> on > >>>>> it > >>>>>>> in > >>>>>>>>> the > >>>>>>>>>>>>> future. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 on this, just ping me. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> After some code digging and POC all I can say that > >> with > >>>>>>> heavy > >>>>>>>>>> effort > >>>>>>>>>>> we > >>>>>>>>>>>>> can > >>>>>>>>>>>>>> maybe add such changes that we're able to show > >> metadata > >>>>> of a > >>>>>>>>>>> savepoint > >>>>>>>>>>>>> from > >>>>>>>>>>>>>> catalog. > >>>>>>>>>>>>>> I'm not against that but from user perspective this > >> has > >>>>>>> limited > >>>>>>>>>>> value, > >>>>>>>>>>>>> let > >>>>>>>>>>>>>> me explain why. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> From high level perspective I see the following > >> which I > >>>>> see > >>>>>>>>>> agreement > >>>>>>>>>>>> on: > >>>>>>>>>>>>>> * We should have a catalog which is representing one > >> or > >>>>> more > >>>>>>>> jobs > >>>>>>>>>>>>> savepoint > >>>>>>>>>>>>>> data set (future plan) > >>>>>>>>>>>>>> * Savepoints should be able to be registered in the > >>>>> catalog > >>>>>>>> which > >>>>>>>>>> are > >>>>>>>>>>>>> then > >>>>>>>>>>>>>> databases (future plan) > >>>>>>>>>>>>>> * There must be a possiblity to create tables from > >>>>> databases > >>>>>>>>> where > >>>>>>>>>>>> users > >>>>>>>>>>>>>> can read state data (exists already) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In terms of metadata, If I understand correctly then > >>>> the > >>>>>>>>> suggested > >>>>>>>>>>>>> approach > >>>>>>>>>>>>>> would be to access > >>>>>>>>>>>>>> it from the catalog describe command, right? Adding > >>>> that > >>>>>>> info > >>>>>>>>> when > >>>>>>>>>>>>> specific > >>>>>>>>>>>>>> database describe command > >>>>>>>>>>>>>> is executed could be done. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The question is for instance how can users create > >> such > >>>> a > >>>>>>> logic > >>>>>>>>> that > >>>>>>>>>>>> tells > >>>>>>>>>>>>>> them what is > >>>>>>>>>>>>>> the difference between multiple savepoints? > >>>>>>>>>>>>>> Just to give some examples: > >>>>>>>>>>>>>> * per operator size changes between savepoints > >>>>>>>>>>>>>> * show values from operator data where state size > >>>> reaches > >>>>> a > >>>>>>>>>> boundary > >>>>>>>>>>>>>> * in general "find which checkpoint ruined things" is > >>>>> quite > >>>>>>>>> common > >>>>>>>>>>>>> pattern > >>>>>>>>>>>>>> What I would like to highlight here is that from > >> Flink > >>>>>>> point of > >>>>>>>>>> view > >>>>>>>>>>>> the > >>>>>>>>>>>>>> metadata can be > >>>>>>>>>>>>>> considered as a static side output information but > >> for > >>>>> users > >>>>>>>>> these > >>>>>>>>>>>> values > >>>>>>>>>>>>>> are actual real data > >>>>>>>>>>>>>> where logic is planned to build around. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The metadata is more like one-time information > >>>> instead > >>>>> of > >>>>>>> a > >>>>>>>>>>> streaming > >>>>>>>>>>>>>> data that changes all > >>>>>>>>>>>>>> the time, so a single connector seems to be an > >>>> overkill. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> State data is also static within a savepoint and > >> that's > >>>>> the > >>>>>>>>> reason > >>>>>>>>>>> why > >>>>>>>>>>>>> the > >>>>>>>>>>>>>> state processor API is working in batch mode. > >>>>>>>>>>>>>> When we handle multiple checkpoints in a streaming > >>>> fashion > >>>>>>> then > >>>>>>>>>> this > >>>>>>>>>>>> can > >>>>>>>>>>>>> be > >>>>>>>>>>>>>> viewed from another angle. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We can come up with more lightweight solution other > >>>> than a > >>>>>>> new > >>>>>>>>>>>> connector > >>>>>>>>>>>>>> but enforcing users to parse the catalog > >>>>>>>>>>>>>> describe command output in order to compare multiple > >>>>>>> savepoints > >>>>>>>>>>> doesn't > >>>>>>>>>>>>>> sound smooth user experience. > >>>>>>>>>>>>>> Honestly I've no other idea how exposing metadata as > >>>> real > >>>>>>> user > >>>>>>>>> data > >>>>>>>>>>> so > >>>>>>>>>>>>>> waiting on other approaches. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>> G > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Mar 13, 2025 at 2:44 AM Shengkai Fang < > >>>>>>>> fskm...@gmail.com > >>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Looking forward to hearing the good news! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> Shengkai > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Gabor Somogyi <gabor.g.somo...@gmail.com> > >>>> 于2025年3月12日周三 > >>>>>>>>> 22:24写道： > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for both the valuable input! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Let me take a closer look at the suggestions, > >> like > >>>> the > >>>>>>>>> Catalog > >>>>>>>>>>>>>>> capabilities > >>>>>>>>>>>>>>>> and possibility of embedding TypeInformation or > >>>>>>>>>>>>>>>> StateDescriptor metadata directly into the raw > >>>> state > >>>>>>>> files... > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>>>> G > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 8:17 AM Shengkai Fang < > >>>>>>>>>> fskm...@gmail.com > >>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks for Zakelly's clarification. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> +1 to delay the discussion about this. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I’d like to share my perspective on the State > >>>>> Catalog > >>>>>>>>>> proposal. > >>>>>>>>>>>>> While > >>>>>>>>>>>>>>>>> introducing this capability is beneficial, > >> there > >>>> is > >>>>> a > >>>>>>>>>> blocker: > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>>> StateBackend architecture does not permit > >>>> operators > >>>>> to > >>>>>>>>> encode > >>>>>>>>>>>>>>>>> TypeInformation into the state—it only > >> preserves > >>>> the > >>>>>>>>>>> Serializer. > >>>>>>>>>>>>> This > >>>>>>>>>>>>>>>>> limitation creates an asymmetry, as operators > >>>> alone > >>>>>>>> retain > >>>>>>>>>>>>> knowledge > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> data structure’s schema. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> To address this, I suggest allowing operators > >> to > >>>>> embed > >>>>>>>>>>>>>> TypeInformation > >>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>> StateDescriptor metadata directly into the raw > >>>> state > >>>>>>>> files. > >>>>>>>>>>> Such > >>>>>>>>>>>> a > >>>>>>>>>>>>>>> design > >>>>>>>>>>>>>>>>> would enable the Catalog to: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. Parse state files and programmatically > >> derive > >>>> the > >>>>>>>> schema > >>>>>>>>>> and > >>>>>>>>>>>>>>>> structural > >>>>>>>>>>>>>>>>> guarantees for each state. > >>>>>>>>>>>>>>>>> 2. Leverage existing Flink Table utilities, > >> such > >>>> as > >>>>>>>>>>>>>>>>> LegacyTypeInfoDataTypeConverter (in > >>>>>>>>>>>>>>> org.apache.flink.table.types.utils), > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> bridge TypeInformation and DataType > >> conversions. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> If we can not store the TypeInformation or > >>>>>>>> StateDescriptor > >>>>>>>>>> into > >>>>>>>>>>>> the > >>>>>>>>>>>>>> raw > >>>>>>>>>>>>>>>>> state files, I am +1 for this FLIP to use > >>>> metadata > >>>>>>> column > >>>>>>>>> to > >>>>>>>>>>>>> retrieve > >>>>>>>>>>>>>>>>> information. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>> Shengkai > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Zakelly Lan <zakelly....@gmail.com> > >>>> 于2025年3月12日周三 > >>>>>>>> 12:43写道： > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi Gabor and Shengkai, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks for sharing your thoughts! This is a > >>>> long > >>>>>>>>> discussion > >>>>>>>>>>> and > >>>>>>>>>>>>>> sorry > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> the late reply (I'm busy catching up with > >>>> release > >>>>>>> 2.0 > >>>>>>>>> these > >>>>>>>>>>>>> days). > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Let me first clarify your thoughts to ensure > >> I > >>>>>>>> understand > >>>>>>>>>>>>>> correctly. > >>>>>>>>>>>>>>>>> IIUC, > >>>>>>>>>>>>>>>>>> there is no persistent configuration for > >> state > >>>> TTL > >>>>>>> in > >>>>>>>> the > >>>>>>>>>>>>>> checkpoint. > >>>>>>>>>>>>>>>>> While > >>>>>>>>>>>>>>>>>> you can infer that TTL is enabled by reading > >>>> the > >>>>>>>>>> serializer, > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> checkpoint > >>>>>>>>>>>>>>>>>> itself only stores the last access time for > >>>> each > >>>>>>> value. > >>>>>>>>> So > >>>>>>>>>>> the > >>>>>>>>>>>>> only > >>>>>>>>>>>>>>>> thing > >>>>>>>>>>>>>>>>>> we can show is the last access time for each > >>>>> value. > >>>>>>> But > >>>>>>>>> it > >>>>>>>>>> is > >>>>>>>>>>>> not > >>>>>>>>>>>>>>>>> required > >>>>>>>>>>>>>>>>>> for all state backends to store this, as they > >>>> may > >>>>>>>>> directly > >>>>>>>>>>>> store > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> expired time. This will also increase the > >>>>>>> difficulty of > >>>>>>>>>>>>>>> implementation > >>>>>>>>>>>>>>>> & > >>>>>>>>>>>>>>>>>> maintenance. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> This once again reiterates the importance of > >>>>> unified > >>>>>>>>>> metadata > >>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> checkpoints. I’m planning on adding this, and > >>>> we > >>>>> may > >>>>>>>>>>>> collaborate > >>>>>>>>>>>>> on > >>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> the future. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I'm not in favor of adding a new connector > >> for > >>>>>>>> metadata. > >>>>>>>>>> The > >>>>>>>>>>>>>> metadata > >>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> more like one-time information instead of a > >>>>>>> streaming > >>>>>>>>> data > >>>>>>>>>>> that > >>>>>>>>>>>>>>> changes > >>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>> the time, so a single connector seems to be > >> an > >>>>>>>> overkill. > >>>>>>>>> It > >>>>>>>>>>> is > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>> easy > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> withdraw a connector if we have a better > >>>> solution > >>>>> in > >>>>>>>>>> future. > >>>>>>>>>>>> I'm > >>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>> familiar with current Catalog capabilities, > >>>> and if > >>>>>>> it > >>>>>>>>> could > >>>>>>>>>>>>> extract > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> show some operator-level information from > >>>>> savepoint, > >>>>>>>> that > >>>>>>>>>>> would > >>>>>>>>>>>>> be > >>>>>>>>>>>>>>>> great. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> If the Catalog can't do that, I would > >> consider > >>>> the > >>>>>>>>> current > >>>>>>>>>>> FLIP > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>> be a > >>>>>>>>>>>>>>>>>> compromise solution. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> And if we have that unified metadata for > >>>>>>>>>> checkpoint/savepoint > >>>>>>>>>>>> in > >>>>>>>>>>>>>>>> future, > >>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>> may directly register savepoint in catalog, > >> and > >>>>>>> create > >>>>>>>> a > >>>>>>>>>>> source > >>>>>>>>>>>>>>> without > >>>>>>>>>>>>>>>>>> specifying complex columns, as well as > >> describe > >>>>> the > >>>>>>>>>> savepoint > >>>>>>>>>>>>>> catalog > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> get the metadata. That's a good solution in > >> my > >>>>> mind. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>> Zakelly > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 10:35 AM Shengkai > >> Fang > >>>> < > >>>>>>>>>>>>> fskm...@gmail.com> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi Gabor, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > >>>>>>> `savepoint-metadata` > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I would argue against introducing a new > >>>>> connector > >>>>>>>> type > >>>>>>>>>>> named > >>>>>>>>>>>>>>>>>>> savepoint-metadata, as the existing Catalog > >>>>>>> mechanism > >>>>>>>>> can > >>>>>>>>>>>>>>> inherently > >>>>>>>>>>>>>>>>>>> provide the necessary connector factory > >>>>>>> capabilities. > >>>>>>>>>> I’ve > >>>>>>>>>>>>>> detailed > >>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>> proposal in branch[1]. Please take a moment > >>>> to > >>>>>>> review > >>>>>>>>> it. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> If we introduce a connector named > >>>>>>>> `savepoint-metadata`, > >>>>>>>>>> it > >>>>>>>>>>>>> means > >>>>>>>>>>>>>>> user > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> create a temporary table with connector > >>>>>>>>>>> `savepoint-metadata` > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> connector needs to check whether table > >>>> schema is > >>>>>>> same > >>>>>>>>> to > >>>>>>>>>>> the > >>>>>>>>>>>>>> schema > >>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>> proposed in the FLIP. On the other hand, > >> it's > >>>>> not > >>>>>>>> easy > >>>>>>>>>> work > >>>>>>>>>>>> for > >>>>>>>>>>>>>>>> others > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> users a metadata table with same schema. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/flink/compare/master...fsk119:flink:state-metadata?expand=1#diff-712a7bc92fe46c405fb0e61b475bb2a005cb7a72bab7df28bbb92744bcb5f465R63 > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>> Shengkai > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Gabor Somogyi <gabor.g.somo...@gmail.com> > >>>>>>>>> 于2025年3月11日周二 > >>>>>>>>>>>>> 16:56写道： > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi Shengkai, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> From directional perspective I agree your > >>>> idea > >>>>>>> how > >>>>>>>> it > >>>>>>>>>> can > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> implemented. > >>>>>>>>>>>>>>>>>>>> Previously I've mentioned that TTL > >>>> information > >>>>>>> is > >>>>>>>> not > >>>>>>>>>>>> exposed > >>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>>> processor API (which the SQL state > >>>> connector > >>>>>>> uses > >>>>>>>> to > >>>>>>>>>> read > >>>>>>>>>>>>> data) > >>>>>>>>>>>>>>>>>>>> and unless somebody show me the opposite > >>>> this > >>>>>>> FLIP > >>>>>>>> is > >>>>>>>>>> not > >>>>>>>>>>>>> going > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> address > >>>>>>>>>>>>>>>>>>>> this to avoid feature creep. Our users > >> are > >>>>> also > >>>>>>>>>>> interested > >>>>>>>>>>>> in > >>>>>>>>>>>>>> TTL > >>>>>>>>>>>>>>>> so > >>>>>>>>>>>>>>>>>>>> sooner or later we're going to expose it, > >>>> this > >>>>>>> is > >>>>>>>>>> matter > >>>>>>>>>>> of > >>>>>>>>>>>>>>>>> scheduling. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > >>>>>>>> `savepoint-metadata` > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Not sure I understand your point at all > >>>>> related > >>>>>>>>>>>> StateCatalog. > >>>>>>>>>>>>>>> First > >>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>> I can't agree more that StateCatalog is > >>>> needed > >>>>>>> and > >>>>>>>>> is a > >>>>>>>>>>>>> planned > >>>>>>>>>>>>>>>>>> building > >>>>>>>>>>>>>>>>>>>> block in an upcoming > >>>>>>>>>>>>>>>>>>>> FLIP but not sure how can it help now? No > >>>>> matter > >>>>>>>>> what, > >>>>>>>>>>> your > >>>>>>>>>>>>>>>> knowledge > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> essential when we add StateCatalog. Let > >> me > >>>>>>> expose > >>>>>>>> my > >>>>>>>>>>>>>>> understanding > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>> area: > >>>>>>>>>>>>>>>>>>>> * First we need create table statements > >> to > >>>>>>> access > >>>>>>>>> state > >>>>>>>>>>>> data > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> metadata > >>>>>>>>>>>>>>>>>>>> * When we have that then we can add > >>>>> StateCatalog > >>>>>>>>> which > >>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>> potentially > >>>>>>>>>>>>>>>>>>>> ease the life of users by for ex. giving > >>>>>>>>> off-the-shelf > >>>>>>>>>>>> tables > >>>>>>>>>>>>>>>> without > >>>>>>>>>>>>>>>>>>>> sweating with create table statements > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> User expectations: > >>>>>>>>>>>>>>>>>>>> * See state data (this is fulfilled with > >>>> the > >>>>>>>> existing > >>>>>>>>>>>>>> connector) > >>>>>>>>>>>>>>>>>>>> * See metadata about state data like TTL > >>>> (this > >>>>>>> can > >>>>>>>> be > >>>>>>>>>>> added > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>> metadata > >>>>>>>>>>>>>>>>>>>> column as you suggested since it belongs > >> to > >>>>> the > >>>>>>>> data) > >>>>>>>>>>>>>>>>>>>> * See metadata about operators (this can > >> be > >>>>>>> added > >>>>>>>>> from > >>>>>>>>>>>>>>>>>>> savepoint-metadata) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Important to highlight that state data > >>>> table > >>>>>>> format > >>>>>>>>>>> differs > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>>> metadata table format. Namely one table > >> has > >>>>> rows > >>>>>>>> for > >>>>>>>>>>> state > >>>>>>>>>>>>>> values > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> another has rows for operators, right? > >>>>>>>>>>>>>>>>>>>> I think that's the reason why you've > >>>>> pinpointed > >>>>>>> out > >>>>>>>>>> that > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> suggested > >>>>>>>>>>>>>>>>>>>> metadata columns are somewhat clunky. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> As a conclusion I agree to add > >>>>> ${state-name}_ttl > >>>>>>>>>> metadata > >>>>>>>>>>>>>> column > >>>>>>>>>>>>>>>>> later > >>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>> since it belongs to the state value and > >>>>> adding a > >>>>>>>> new > >>>>>>>>>>> table > >>>>>>>>>>>>> type > >>>>>>>>>>>>>>>> (like > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>> suggested similar to PG [1]) > >>>>>>>>>>>>>>>>>>>> for metadata. Please see how Spark does > >>>> that > >>>>> too > >>>>>>>> [2]. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> If you have better approach then please > >>>>>>> elaborate > >>>>>>>>> with > >>>>>>>>>>> more > >>>>>>>>>>>>>>> details > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> help me to understand your point. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB > >>>>> savepoints > >>>>>>>> that > >>>>>>>>>> the > >>>>>>>>>>>>> number > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> keys > >>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key > >>>> state > >>>>>>>> itself. > >>>>>>>>>>>>>>>>>>>>> But again, this is a good feature as-is > >>>> and > >>>>>>> can > >>>>>>>> be > >>>>>>>>>>>> handled > >>>>>>>>>>>>>> in a > >>>>>>>>>>>>>>>>>>> separate > >>>>>>>>>>>>>>>>>>>>> jira. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I've just created > >>>>>>>>>>>>>>>> > >> https://issues.apache.org/jira/browse/FLINK-37456. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>> > >>>>> https://www.postgresql.org/docs/current/view-pg-tables.html > >>>>>>>>>>>>>>>>>>>> [2] > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>>>>>>>> G > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Tue, Mar 11, 2025 at 3:55 AM Shengkai > >>>> Fang > >>>>> < > >>>>>>>>>>>>>> fskm...@gmail.com > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your response. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thank you for addressing the > >> limitations > >>>>> here. > >>>>>>>>>>> However, I > >>>>>>>>>>>>>>> believe > >>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>> be beneficial to further clarify the > >> API > >>>> in > >>>>>>> this > >>>>>>>>> FLIP > >>>>>>>>>>>>>> regarding > >>>>>>>>>>>>>>>> how > >>>>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>>>>> can specify the TTL column. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> One potential approach that comes to > >>>> mind is > >>>>>>>> using > >>>>>>>>> a > >>>>>>>>>>>>>>> standardized > >>>>>>>>>>>>>>>>>>> naming > >>>>>>>>>>>>>>>>>>>>> convention such as ${state-name}_ttl > >> for > >>>> the > >>>>>>>>> metadata > >>>>>>>>>>>>> column > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>> defines > >>>>>>>>>>>>>>>>>>>>> the TTL value. In terms of > >>>> implementation, > >>>>> the > >>>>>>>>>>>>>>>> listReadableMetadata > >>>>>>>>>>>>>>>>>>>>> function could: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 1. Read the table’s columns and > >>>>> configuration, > >>>>>>>>>>>>>>>>>>>>> 2. Extract all defined state names, and > >>>>>>>>>>>>>>>>>>>>> 3. Return a structured list of metadata > >>>>>>> entries > >>>>>>>>>>> formatted > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>> ${state-name}_ttl. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> WDYT? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with > >>>>>>>>> `savepoint-metadata` > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Introducing a new connector type at > >> this > >>>>> stage > >>>>>>>> may > >>>>>>>>>>>>>>> unnecessarily > >>>>>>>>>>>>>>>>>>>> complicate > >>>>>>>>>>>>>>>>>>>>> the system. Given that every table > >>>> already > >>>>>>>> belongs > >>>>>>>>>> to a > >>>>>>>>>>>>>>> Catalog, > >>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>> designed to provide a Factory for > >>>> building > >>>>>>> source > >>>>>>>>> or > >>>>>>>>>>> sink > >>>>>>>>>>>>>>>>>> connectors, I > >>>>>>>>>>>>>>>>>>>>> propose integrating a dedicated > >>>> StateCatalog > >>>>>>>>> instead. > >>>>>>>>>>>> This > >>>>>>>>>>>>>>>> approach > >>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>> allow us to: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 1. Leverage the Catalog’s existing > >>>>>>> capabilities > >>>>>>>> to > >>>>>>>>>>> manage > >>>>>>>>>>>>> TTL > >>>>>>>>>>>>>>>>>> metadata > >>>>>>>>>>>>>>>>>>>>> (e.g., state names and TTL logic) > >> without > >>>>>>>>> duplicating > >>>>>>>>>>>>>>>>> functionality. > >>>>>>>>>>>>>>>>>>>>> 2. Provide a unified interface for > >>>> connector > >>>>>>>>>>>> instantiation > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> metadata > >>>>>>>>>>>>>>>>>>>>> handling through the Catalog’s Factory > >>>>>>> pattern. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Would this design decision better align > >>>> with > >>>>>>> our > >>>>>>>>>>>>>> architecture’s > >>>>>>>>>>>>>>>>>>>>> extensibility and reduce redundancy? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB > >>>>>>> savepoints > >>>>>>>>> that > >>>>>>>>>>> the > >>>>>>>>>>>>>> number > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> keys > >>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key > >>>>> state > >>>>>>>>> itself. > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature > >> as-is > >>>>> and > >>>>>>> can > >>>>>>>>> be > >>>>>>>>>>>>> handled > >>>>>>>>>>>>>>> in a > >>>>>>>>>>>>>>>>>>>> separate > >>>>>>>>>>>>>>>>>>>>>> jira. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> +1 for a separate jira. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>> Shengkai > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < > >> gabor.g.somo...@gmail.com > >>>>> > >>>>>>>>>>> 于2025年3月10日周一 > >>>>>>>>>>>>>>> 19:05写道： > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hi Shengkai, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Please see my comments inline. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>>>>>>>>>> G > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 3, 2025 at 7:07 AM > >> Shengkai > >>>>>>> Fang < > >>>>>>>>>>>>>>>> fskm...@gmail.com> > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your the > >> FLIP. > >>>> I > >>>>>>> have > >>>>>>>>> some > >>>>>>>>>>>>>> questions > >>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> FLIP: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns > >>>>>>>>>>>>>>>>>>>>>>> How can users retrieve the state > >> TTL > >>>>>>>>>> (Time-to-Live) > >>>>>>>>>>>> for > >>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>>> value > >>>>>>>>>>>>>>>>>>>>>> column? > >>>>>>>>>>>>>>>>>>>>>>> From my understanding of the > >> current > >>>>>>> design, > >>>>>>>> it > >>>>>>>>>>> seems > >>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>> functionality is not supported. > >> Could > >>>>> you > >>>>>>>>> clarify > >>>>>>>>>>> if > >>>>>>>>>>>>>> there > >>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>> plans > >>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> address this limitation? > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Since the state processor API is not > >>>> yet > >>>>>>>> exposing > >>>>>>>>>>> this > >>>>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>> would require several steps. > >>>>>>>>>>>>>>>>>>>>>> First, the state processor API > >> support > >>>>>>> needs to > >>>>>>>>> be > >>>>>>>>>>>> added > >>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>>>>>> exposed on the SQL API. > >>>>>>>>>>>>>>>>>>>>>> This is definitely a future > >> improvement > >>>>>>> which > >>>>>>>> is > >>>>>>>>>>> useful > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>> handled > >>>>>>>>>>>>>>>>>>>>>> in a separate jira. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata > >> Column > >>>>>>>>>>>>>>>>>>>>>>> The metadata information described > >> in > >>>>> the > >>>>>>>> FLIP > >>>>>>>>>>>> appears > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>> intended > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> describe the state files stored at > >> a > >>>>>>> specific > >>>>>>>>>>>> location. > >>>>>>>>>>>>>> To > >>>>>>>>>>>>>>>> me, > >>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>> concept > >>>>>>>>>>>>>>>>>>>>>>> aligns more closely with system > >>>> tables > >>>>>>> like > >>>>>>>>>>> pg_tables > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> PostgreSQL > >>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>>>> the INFORMATION_SCHEMA in MySQL > >> [2]. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Adding a new connector with > >>>>>>>> `savepoint-metadata` > >>>>>>>>>> is a > >>>>>>>>>>>>>>>> possibility > >>>>>>>>>>>>>>>>>>> where > >>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>> can create such functionality. > >>>>>>>>>>>>>>>>>>>>>> I'm not against that, just want to > >>>> have a > >>>>>>>> common > >>>>>>>>>>>>> agreement > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> like to move that direction. > >>>>>>>>>>>>>>>>>>>>>> (As a side note not just PG but Spark > >>>> also > >>>>>>> has > >>>>>>>>>>> similar > >>>>>>>>>>>>>>> approach > >>>>>>>>>>>>>>>>>> and I > >>>>>>>>>>>>>>>>>>>>>> basically like the idea). > >>>>>>>>>>>>>>>>>>>>>> If we would go that direction > >> savepoint > >>>>>>>> metadata > >>>>>>>>>> can > >>>>>>>>>>> be > >>>>>>>>>>>>>>> reached > >>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>> way > >>>>>>>>>>>>>>>>>>>>>> that one row would represent > >>>>>>>>>>>>>>>>>>>>>> an operator with it's values > >> something > >>>>> like > >>>>>>>> this: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┐ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > │operatorN│operatorU│operatorH│paralleli│maxParall│subtaskSt│coordinat│totalSta│ > >>>>>>>>>>>>>>>>>>>>>> │ame │id │ash │sm > >>>>>>> │elism > >>>>>>>>>>>>>>>>>>>>>> │atesCount│orStateSi│tesSizeI│ > >>>>>>>>>>>>>>>>>>>>>> │ │ │ │ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │zeInBytes│nBytes │ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > >>>>>>>>>>>>>>>>>>>>>> │Source: │datagen-s│47aee9439│2 > >>>>> │128 > >>>>>>>>>> │2 > >>>>>>>>>>>>>>> │16 > >>>>>>>>>>>>>>>>>>>>>> │546 │ > >>>>>>>>>>>>>>>>>>>>>> │datagen-s│ource-uid│4d6ea26e2│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ource │ │d544bef0a│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ │ │37bb5 │ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > >>>>>>>>>>>>>>>>>>>>>> │long-udf-│long-udf-│6ed3f40bf│2 > >>>>> │128 > >>>>>>>>>> │2 > >>>>>>>>>>>>>>> │0 > >>>>>>>>>>>>>>>>>>>> │0 > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │with-mast│with-mast│f3c8dfcdf│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │er-hook │er-hook-u│cb95128a1│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ │id │018f1 │ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > >>>>>>>>>>>>>>>>>>>>>> │value-pro│value-pro│ca4f5fe9a│2 > >>>>> │128 > >>>>>>>>>> │2 > >>>>>>>>>>>>>>> │0 > >>>>>>>>>>>>>>>>>>>>>> │40726 │ > >>>>>>>>>>>>>>>>>>>>>> │cess │cess-uid │637b656f0│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ │ │9ea78b3e7│ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ │ │a15b9 │ > >>>> │ > >>>>>>>>> │ > >>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> │ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> This table can then be joined with > >> the > >>>>>>> actually > >>>>>>>>>>>> existing > >>>>>>>>>>>>>>>>>> `savepoint` > >>>>>>>>>>>>>>>>>>>>>> connector created tables based on UID > >>>> hash > >>>>>>>> (which > >>>>>>>>>> is > >>>>>>>>>>>>> unique > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> always > >>>>>>>>>>>>>>>>>>>>>> exists). > >>>>>>>>>>>>>>>>>>>>>> This would mean that the already > >>>> existing > >>>>>>> table > >>>>>>>>>> would > >>>>>>>>>>>>> need > >>>>>>>>>>>>>>>> only a > >>>>>>>>>>>>>>>>>>>> single > >>>>>>>>>>>>>>>>>>>>>> metadata column which is the UID > >> hash. > >>>>>>>>>>>>>>>>>>>>>> WDYT? > >>>>>>>>>>>>>>>>>>>>>> @zakelly, plz share your thoughts > >> too. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> If we opt to use metadata columns, > >>>> every > >>>>>>>> record > >>>>>>>>>> in > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> table > >>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>>>>>>> up > >>>>>>>>>>>>>>>>>>>>>>> having identical values for these > >>>>> columns > >>>>>>>>> (please > >>>>>>>>>>>>> correct > >>>>>>>>>>>>>>> me > >>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>> I’m > >>>>>>>>>>>>>>>>>>>>>>> mistaken). On the other hand, the > >>>> state > >>>>>>>>> connector > >>>>>>>>>>>>>> requires > >>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> specify > >>>>>>>>>>>>>>>>>>>>>>> an operator UID or operator UID > >> hash, > >>>>>>> after > >>>>>>>>> which > >>>>>>>>>>> it > >>>>>>>>>>>>>>> outputs > >>>>>>>>>>>>>>>>>>>>> user-defined > >>>>>>>>>>>>>>>>>>>>>>> values in its records. This > >> approach > >>>>> feels > >>>>>>>>>> somewhat > >>>>>>>>>>>>>>> redundant > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> me. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> If we would add a new > >>>> `savepoint-metadata` > >>>>>>>>>> connector > >>>>>>>>>>>> then > >>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>> addressed. > >>>>>>>>>>>>>>>>>>>>>> On the other hand UID and UID hash > >> are > >>>>>>> having > >>>>>>>>>>> either-or > >>>>>>>>>>>>>>>>>> relationship > >>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>> config perspective, > >>>>>>>>>>>>>>>>>>>>>> so when a user provides the UID then > >>>>> he/she > >>>>>>> can > >>>>>>>>> be > >>>>>>>>>>>>>> interested > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> hash > >>>>>>>>>>>>>>>>>>>>>> for further calculations > >>>>>>>>>>>>>>>>>>>>>> (the whole Flink internals are > >>>> depending > >>>>> on > >>>>>>> the > >>>>>>>>>>> hash). > >>>>>>>>>>>>>>> Printing > >>>>>>>>>>>>>>>>> out > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> human readable UID > >>>>>>>>>>>>>>>>>>>>>> is an explicit requirement from the > >>>> user > >>>>>>> side > >>>>>>>>>> because > >>>>>>>>>>>>>> hashes > >>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>> human > >>>>>>>>>>>>>>>>>>>>>> readable. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 3. Handling LIST and MAP States in > >>>> the > >>>>>>> State > >>>>>>>>>>>> Connector > >>>>>>>>>>>>>>>>>>>>>>> I have concerns about how the > >> current > >>>>>>> design > >>>>>>>>>>> handles > >>>>>>>>>>>>> LIST > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> MAP > >>>>>>>>>>>>>>>>>>>>> states. > >>>>>>>>>>>>>>>>>>>>>>> Specifically, the state connector > >>>> uses > >>>>>>> Flink > >>>>>>>>>> SQL’s > >>>>>>>>>>>> MAP > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> ARRAY > >>>>>>>>>>>>>>>>>>>> types, > >>>>>>>>>>>>>>>>>>>>>>> which implies that it attempts to > >>>> load > >>>>>>> entire > >>>>>>>>> MAP > >>>>>>>>>>> or > >>>>>>>>>>>>> LIST > >>>>>>>>>>>>>>>>> states > >>>>>>>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>>>>>>>>> memory. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> However, in many real-world > >>>> scenarios, > >>>>>>> these > >>>>>>>>>> states > >>>>>>>>>>>> can > >>>>>>>>>>>>>>> grow > >>>>>>>>>>>>>>>>> very > >>>>>>>>>>>>>>>>>>>>> large. > >>>>>>>>>>>>>>>>>>>>>>> Typically, the state API addresses > >>>> this > >>>>> by > >>>>>>>>>>> providing > >>>>>>>>>>>> an > >>>>>>>>>>>>>>>>> iterator > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> traverse elements within the state > >>>>>>>>> incrementally. > >>>>>>>>>>> I’m > >>>>>>>>>>>>>>> unsure > >>>>>>>>>>>>>>>>>>> whether > >>>>>>>>>>>>>>>>>>>>> I’ve > >>>>>>>>>>>>>>>>>>>>>>> missed something in FLIP-496 or > >>>>> FLIP-512, > >>>>>>> but > >>>>>>>>> it > >>>>>>>>>>>> seems > >>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>>>>>>>>> design might struggle with > >>>> scalability > >>>>> in > >>>>>>>> such > >>>>>>>>>>> cases. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> You see it good, the current > >>>>> implementation > >>>>>>>> keeps > >>>>>>>>>>> state > >>>>>>>>>>>>>> for a > >>>>>>>>>>>>>>>>>> single > >>>>>>>>>>>>>>>>>>>> key > >>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>> memory. > >>>>>>>>>>>>>>>>>>>>>> Back in the days we've considered > >> this > >>>>>>>> potential > >>>>>>>>>>> issue > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> concluded > >>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>> this is not necessarily > >>>>>>>>>>>>>>>>>>>>>> needed for the initial version and > >> can > >>>> be > >>>>>>> done > >>>>>>>>> as a > >>>>>>>>>>>> later > >>>>>>>>>>>>>>>>>>> improvement. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB > >>>>>>> savepoints > >>>>>>>>> that > >>>>>>>>>>> the > >>>>>>>>>>>>>> number > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> keys > >>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key > >>>>> state > >>>>>>>>> itself. > >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature > >> as-is > >>>>> and > >>>>>>> can > >>>>>>>>> be > >>>>>>>>>>>>> handled > >>>>>>>>>>>>>>> in a > >>>>>>>>>>>>>>>>>>>> separate > >>>>>>>>>>>>>>>>>>>>>> jira. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>> Shengkai > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>> > >>>>>>>>> https://www.postgresql.org/docs/current/view-pg-tables.html > >>>>>>>>>>>>>>>>>>>>>>> [2] > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < > >>>>> gabor.g.somo...@gmail.com> > >>>>>>>>>>>> 于2025年3月3日周一 > >>>>>>>>>>>>>>>>> 02:00写道： > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Zakelly, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> In order to shoot for simplicity > >>>>>>> `METADATA > >>>>>>>>>>> VIRTUAL` > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>> key > >>>>>>>>>>>>>>>>>> words > >>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>> definition is the target. > >>>>>>>>>>>>>>>>>>>>>>>> When it's not super complex the > >>>> latter > >>>>>>> can > >>>>>>>> be > >>>>>>>>>>> added > >>>>>>>>>>>>>> too. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>>>>>>>>>>>> G > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 2, 2025 at 3:37 PM > >>>> Zakelly > >>>>>>> Lan > >>>>>>>> < > >>>>>>>>>>>>>>>>>>> zakelly....@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Gabor, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> +1 for this. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Will the metadata column use > >>>>> `METADATA > >>>>>>>>>> VIRTUAL` > >>>>>>>>>>>> as > >>>>>>>>>>>>>> key > >>>>>>>>>>>>>>>>> words > >>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>> definition, or `METADATA FROM > >> xxx > >>>>>>>> VIRTUAL` > >>>>>>>>>> for > >>>>>>>>>>>>>>> renaming, > >>>>>>>>>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> Kafka table? > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>>> Zakelly > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Mar 1, 2025 at 1:31 PM > >>>> Gabor > >>>>>>>>> Somogyi > >>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> gabor.g.somo...@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start a > >> discussion > >>>> of > >>>>>>>>> FLIP-512: > >>>>>>>>>>> Add > >>>>>>>>>>>>>> meta > >>>>>>>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> SQL > >>>>>>>>>>>>>>>>>>>>>>>>>> state connector [1]. > >>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to add your > >> thoughts > >>>> to > >>>>>>> make > >>>>>>>>> this > >>>>>>>>>>>>> feature > >>>>>>>>>>>>>>>>> better. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> BR, > >>>>>>>>>>>>>>>>>>>>>>>>>> G > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
