Your link is broken, Shengkai Best, Leonard
> 2025年3月27日 10:01，Shengkai Fang <fskm...@gmail.com> 写道： > > Hi, All. > > I write a simple demo to illustrate my idea. Hope this helps. > > Best, > Shengkai > > https://github.com/apache/flink/compare/master...fsk119:flink:example?expand=1 > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月26日周三 15:54写道： > >>> I'm fine with a seperate SQL connector for metadata, so maybe we could >> update the FLIP about our discussion? >> >> Sorry, I've forgotten this part. Yeah, no matter we choose I'm going to >> update the FLIP. >> >> G >> >> >> On Wed, Mar 26, 2025 at 8:51 AM Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> Hi All, >>> >>> I've also lack of the knowledge of PTF so I've read just the motivation >>> part: >>> >>> "The SQL 2016 standard introduced a way of defining custom SQL operators >>> defined by ISO/IEC 19075-7:2021 (Part 7: Polymorphic table functions). >>> ~200 pages define how this new kind of function can consume and produce >>> tables with various execution properties. >>> Unfortunately, this part of the standard is not publicly available." >>> >>> Of course we can take a look at some examples but do we really want to >>> expose state data with this construct >>> which is described in ~200 pages and part of the standard is not publicly >>> available? 🙂 >>> I mean the dataset is couple of rows and the use-case is join with >> another >>> table like with state data. >>> If somebody can give advantages I would buy that but from my limited >>> understanding this would be an overkill here. >>> >>> BR, >>> G >>> >>> >>> On Wed, Mar 26, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> wrote: >>> >>>> Hi Zakelly , Shengkai! >>>> >>>> I don't know too much about PTFs, it would be interesting to see how the >>>> usage would look in practice. >>>> >>>> Do you have some mockup/example in mind how the PTF would look for >> example >>>> when want to: >>>> - Simply display/aggregate whats in the metadata >>>> - Join keyed state with some metadata columns >>>> >>>> Thanks >>>> Gyula >>>> >>>> On Wed, Mar 26, 2025 at 7:33 AM Zakelly Lan <zakelly....@gmail.com> >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I'm fine with a seperate SQL connector for metadata, so maybe we could >>>>> update the FLIP about our discussion? And Shengkai provides a PTF >>>>> implementation, does that also meet the requirement? >>>>> >>>>> >>>>> Best, >>>>> Zakelly >>>>> >>>>> On Thu, Mar 20, 2025 at 4:47 PM Gabor Somogyi < >>>> gabor.g.somo...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> @Zakelly: Gyula summarised it correctly what I meant so please treat >>>> the >>>>>> content as mine. >>>>>> As an addition I'm not against to add CLI at all, I'm just stating >>>> that >>>>> in >>>>>> some cases like this, users would like to have >>>>>> a self-serving solution where they can provide SQL statements which >>>> can >>>>>> trigger alerts automatically. >>>>>> >>>>>> My personal opinion is that CLI would be beneficial for several >>>> cases. A >>>>>> good example is when users want to restart job >>>>>> from specific Kafka offsets which are persisted in a savepoint. For >>>> such >>>>>> scenario users are more than happy since they >>>>>> expect manual intervention with full control. So all in all one can >>>> count >>>>>> on my +1 when CLI FLIP would come up... >>>>>> >>>>>> BR, >>>>>> G >>>>>> >>>>>> >>>>>> On Thu, Mar 20, 2025 at 8:20 AM Gyula Fóra <gyula.f...@gmail.com> >>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> @Zakelly Lan <zakelly....@gmail.com> >>>>>>> I think what Gabor means is that users want to have predefined SQL >>>>> scripts >>>>>>> to perform state analysis tasks to debug/identify problems. >>>>>>> Such as write a SQL script that joins the metadata table with the >>>> state >>>>>>> and >>>>>>> do some analytics on it. >>>>>>> >>>>>>> If we have a meta table then the SQL script that can do this is >> fixed >>>>> and >>>>>>> users can trigger this on demand by simply providing a new >> savepoint >>>>> path. >>>>>>> >>>>>>> If we have a different mechanism to extract metadata that is not >> SQL >>>>>>> native >>>>>>> then manual steps need to be executed and a custom SQL script would >>>> need >>>>>>> to >>>>>>> be written that adds the manually extracted metadata into the >> script. >>>>>>> >>>>>>> Cheers, >>>>>>> Gyula >>>>>>> >>>>>>> On Thu, Mar 20, 2025 at 4:32 AM Zakelly Lan <zakelly....@gmail.com >>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Thanks for your answers! Getting everyone aligned on this topic >> is >>>>>>>> challenging, but it’s definitely worth the effort since it will >>>> help >>>>>>>> streamline things moving forward. >>>>>>>> >>>>>>>> @Gabor are you saying that users are using some scripts to define >>>> the >>>>>>> SQL >>>>>>>> metadata connector and get the information, right? If so, would a >>>> CLI >>>>>>> tool >>>>>>>> be more convenient? It's easy to invoke and can get the result >>>>> swiftly. >>>>>>> And >>>>>>>> there should be some other systems to track the checkpoint >> lineage >>>> and >>>>>>>> analyze if there are outliers in metadata (e.g. state size of one >>>>>>> operator) >>>>>>>> right? Well, maybe I missed something so please correct me if I'm >>>>> wrong. >>>>>>>> >>>>>>>> I think the overall vision in Flink SQL is to provide a SQL >> native >>>>>>>>> environment where we can serve complex use-cases like you would >>>>> expect >>>>>>>> in a >>>>>>>>> regular database. >>>>>>>> >>>>>>>> >>>>>>>> @Gyula Well, this is a good point. From the perspective of >>>>> comprehensive >>>>>>>> SQL experience, I'd +1 for treating metadata as data. Although I >>>> doubt >>>>>>> if >>>>>>>> there is a need for processing metadata, I won't be against a >>>> separate >>>>>>>> connector. >>>>>>>> >>>>>>>> Regarding the CLI tool, I still think it’s worth implementing. >>>> Such a >>>>>>> tool >>>>>>>> could provide savepoint information before resuming from a >>>> savepoint, >>>>>>> which >>>>>>>> would enhance the user experience in CLI-based workflows. It >> would >>>> be >>>>>>> good >>>>>>>> if someone could implement this feature. We shouldn’t worry about >>>>>>> whether >>>>>>>> this tool might be retired in the future. Regardless of the >>>> SQL-based >>>>>>>> solution we eventually adopt, this capability will remain >> essential >>>>> for >>>>>>> CLI >>>>>>>> users. This is another topic. >>>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> Zakelly >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 20, 2025 at 10:37 AM Shengkai Fang < >> fskm...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi. >>>>>>>>> >>>>>>>>> After reading the doc[1], I think Spark provides a function for >>>>> users >>>>>>> to >>>>>>>>> consume the metadata from the savepoint. In Flink SQL, similar >>>>>>>>> functionality is implemented through Polymorphic Table >> Functions >>>>>>> (PTF) as >>>>>>>>> proposed in FLIP-440[2]. Below is a code example[3] >> illustrating >>>>> this >>>>>>>>> concept: >>>>>>>>> >>>>>>>>> ``` >>>>>>>>> public static class ScalarArgsFunction extends >>>>>>>>> TestProcessTableFunctionBase { >>>>>>>>> public void eval(Integer i, Boolean b) { >>>>>>>>> collectObjects(i, b); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> ``` >>>>>>>>> >>>>>>>>> ``` >>>>>>>>> INSERT INTO sink SELECT * FROM f(i => 42, b => CAST('TRUE' AS >>>>>>> BOOLEAN)) >>>>>>>>> `` >>>>>>>>> >>>>>>>>> So we can add a builtin function named `read_state_metadata` to >>>> read >>>>>>>>> savepoint data. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Shengkai >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://docs.databricks.com/aws/en/structured-streaming/read-state?language=SQL >>>>>>>>> [2] >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=298781093 >>>>>>>>> [3] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java#L140 >>>>>>>>> >>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月19日周三 18:37写道： >>>>>>>>> >>>>>>>>>> Hi All! >>>>>>>>>> >>>>>>>>>> Thank you for the answers and concerns from everyone. >>>>>>>>>> >>>>>>>>>> On the CLI vs State Metadata Connector/Table question I would >>>> also >>>>>>> like >>>>>>>>> to >>>>>>>>>> step back a little and look at the bigger picture. >>>>>>>>>> >>>>>>>>>> I think the overall vision in Flink SQL is to provide a SQL >>>> native >>>>>>>>>> environment where we can serve complex use-cases like you >> would >>>>>>> expect >>>>>>>>> in a >>>>>>>>>> regular database. >>>>>>>>>> Most features, developments in the recent years have gone >> this >>>>> way. >>>>>>>>>> >>>>>>>>>> The State Metadata Table would be a natural and >> straightforward >>>>> fit >>>>>>>> here. >>>>>>>>>> So from my side, +1 for that. >>>>>>>>>> >>>>>>>>>> However I could understand if we are not ready to add a new >>>>>>>>>> connector/format due to maintenance concerns (and in general >>>>> concern >>>>>>>>> about >>>>>>>>>> the design). >>>>>>>>>> If that's the issue then we should spend more time on the >>>> design >>>>> to >>>>>>> get >>>>>>>>>> comfortable with the approach and seek feedback from the >> wider >>>>>>>> community >>>>>>>>>> >>>>>>>>>> I am -1 for the CLI/tooling approach as that will not provide >>>> the >>>>>>>>>> featureset we are looking for that is not already covered by >>>> the >>>>>>> Java >>>>>>>>>> connector. And that approach would come with the same >>>> maintenance >>>>>>>>>> implications. >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> Gyula >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Mar 19, 2025 at 11:24 AM Gabor Somogyi < >>>>>>>>> gabor.g.somo...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Zaklely, Shengkai >>>>>>>>>>> >>>>>>>>>>> Several topics are going on so adding gist answers to them. >>>> When >>>>>>> some >>>>>>>>>> topic >>>>>>>>>>> is not touched please highlight it. >>>>>>>>>>> >>>>>>>>>>> @Shengkai: I've read through all the previous FLIPs related >>>>>>> catalogs >>>>>>>>> and >>>>>>>>>> if >>>>>>>>>>> we would like to keep the concepts there >>>>>>>>>>> then one-to-one mapping relationship between savepoint and >>>>> catalog >>>>>>>> is a >>>>>>>>>>> reasonable direction. In short I'm happy that >>>>>>>>>>> you've highlighted this and agree as a whole. I've written >> it >>>>> down >>>>>>>>>>> previously, just want to double confirm that state catalog >> is >>>>>>>>>>> essential and planned. When we reach this point then your >>>> input >>>>> is >>>>>>>> more >>>>>>>>>>> than welcome. >>>>>>>>>>> >>>>>>>>>>> @Zakelly: We've tried the CLI and separate library >> approaches >>>>> with >>>>>>>>> users >>>>>>>>>>> already and these are not something which is welcome >> because >>>> of >>>>>>> the >>>>>>>>>>> following: >>>>>>>>>>> * Users want to have automated tasks and not manual >>>> CLI/library >>>>>>>> output >>>>>>>>>>> parsing. This can be hacked around but our experience is >>>>> negative >>>>>>> on >>>>>>>>> this >>>>>>>>>>> because it's just brittle. >>>>>>>>>>> * From development perspective It's way much bigger effort >>>> than >>>>> a >>>>>>>>>> connector >>>>>>>>>>> (hard to test, packaging/version handling is and extra >> layer >>>> of >>>>>>>>>> complexity, >>>>>>>>>>> external FS authentication is pain for users, expecting >> them >>>> to >>>>>>>>> download >>>>>>>>>>> savepoints also) >>>>>>>>>>> * Purely personal opinion but if we would find better ways >>>> later >>>>>>> then >>>>>>>>>>> retire a CLI is not more lightweight than retire a >> connector >>>>>>>>>>> >>>>>>>>>>>> It would be great if you give some examples on how user >>>> could >>>>>>>>> leverage >>>>>>>>>>> the separate connector to process the metadata. >>>>>>>>>>> >>>>>>>>>>> The most simplest cases: >>>>>>>>>>> * give me the overgroving state uids >>>>>>>>>>> * give me the not known (new or renamed) state uids >>>>>>>>>>> * give me the state uids where state size drastically >> dropped >>>>>>> compare >>>>>>>>> to >>>>>>>>>> a >>>>>>>>>>> previous savepoint (accidental state loss) >>>>>>>>>>> >>>>>>>>>>> Since it was mentioned: as a general offtopic teaser, yeah >> it >>>>>>> would >>>>>>>> be >>>>>>>>>> good >>>>>>>>>>> to have some sort of checkpoint/savepoint lineage or >> however >>>> we >>>>>>> call >>>>>>>>> it. >>>>>>>>>>> Since we've not yet reached this point there are no >> technical >>>>>>>> details, >>>>>>>>>> it's >>>>>>>>>>> more like a vision. It's a common pattern that >>>>>>>>>>> jobs are physically running but somehow the state >> processing >>>> is >>>>>>> stuck >>>>>>>>> and >>>>>>>>>>> it would be good to add some way to find it out >>>> automatically. >>>>>>>>>>> The important saying here is automation and not manual >>>>> evaluation >>>>>>>> since >>>>>>>>>>> handling 10k+ jobs is just not allowing that. >>>>>>>>>>> >>>>>>>>>>> BR, >>>>>>>>>>> G >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 19, 2025 at 6:46 AM Shengkai Fang < >>>>> fskm...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, All. >>>>>>>>>>>> >>>>>>>>>>>> About State Catalog, I want to share more thoughts about >>>> this. >>>>>>>>>>>> >>>>>>>>>>>> In the initial design concept, I understood that a >>>> savepoint >>>>>>> and a >>>>>>>>>> state >>>>>>>>>>>> catalog have a one-to-one mapping relationship. Each >>>> operator >>>>>>>>>> corresponds >>>>>>>>>>>> to a database, and the state of each operator is >>>> represented >>>>> as >>>>>>>>>>> individual >>>>>>>>>>>> tables. The rationale behind this design is: >>>>>>>>>>>> >>>>>>>>>>>> *State Diversity*: An operator may involve multiple types >>>> of >>>>>>>> states. >>>>>>>>>> For >>>>>>>>>>>> example, in our VVR design, a "multi-join" operator uses >>>> keyed >>>>>>>> states >>>>>>>>>> for >>>>>>>>>>>> two input streams and a broadcast state for the third >>>> stream. >>>>>>> This >>>>>>>>>> makes >>>>>>>>>>> it >>>>>>>>>>>> challenging to represent all states of an operator >> within a >>>>>>> single >>>>>>>>>> table. >>>>>>>>>>>> *Scalability*: Internally, an operator might have >> multiple >>>>> keyed >>>>>>>>> states >>>>>>>>>>>> (e.g., value state and list state). However, large list >>>> states >>>>>>> may >>>>>>>>> not >>>>>>>>>>> fit >>>>>>>>>>>> entirely in memory. To address this, we recommend >>>> implementing >>>>>>> each >>>>>>>>>> state >>>>>>>>>>>> as a separate table. >>>>>>>>>>>> >>>>>>>>>>>> To resolve the loosely coupled relationships between >>>> operator >>>>>>>> states, >>>>>>>>>> we >>>>>>>>>>>> propose embedding predefined views within the catalog. >>>> These >>>>>>> views >>>>>>>>>>> simplify >>>>>>>>>>>> user understanding of operator implementations and >> provide >>>> a >>>>>>> more >>>>>>>>>>> intuitive >>>>>>>>>>>> perspective. For instance, a join operator may have >>>> multiple >>>>>>> state >>>>>>>>>>>> implementations (depending on whether the join key >> includes >>>>>>> unique >>>>>>>>>>>> attributes), but users primarily care about the data >>>>> associated >>>>>>>> with >>>>>>>>> a >>>>>>>>>>>> specific join key across input streams. >>>>>>>>>>>> >>>>>>>>>>>> Returning to the one-to-one mapping between savepoints >> and >>>>>>>> catalogs, >>>>>>>>> we >>>>>>>>>>> aim >>>>>>>>>>>> to manage multiple user state catalogs through a catalog >>>>> store. >>>>>>>> When >>>>>>>>> a >>>>>>>>>>> user >>>>>>>>>>>> triggers a savepoint for a job on the platform: >>>>>>>>>>>> >>>>>>>>>>>> 1. The platform sends a REST request to the JobManager. >>>>>>>>>>>> 2. Simultaneously, it registers a new state catalog in >> the >>>>>>> catalog >>>>>>>>>> store, >>>>>>>>>>>> enabling immediate analysis of state data on the >> platform. >>>>>>>>>>>> 3. Deleting a savepoint would also trigger the removal of >>>> its >>>>>>>>>> associated >>>>>>>>>>>> catalog. >>>>>>>>>>>> >>>>>>>>>>>> This vision assumes that states are self-describing or >>>> that a >>>>>>> state >>>>>>>>>>>> metaservice is introduced to analyze savepoint >> structures. >>>>>>>>>>>> >>>>>>>>>>>>> How can users create logic to identify differences >>>> between >>>>>>>> multiple >>>>>>>>>>>> savepoints? >>>>>>>>>>>> >>>>>>>>>>>> Since savepoints and state catalogs are one-to-one >> mapped, >>>>> users >>>>>>>> can >>>>>>>>>>> query >>>>>>>>>>>> metadata via their respective catalogs. For example: >>>>>>>>>>>> >>>>>>>>>>>> 1. >>>>> `savepoint-${id}`.`system`.`metadata_table`.`<operator-name>` >>>>>>>>>> provides >>>>>>>>>>>> operator-specific metadata (e.g., state size, type). >>>>>>>>>>>> 2. Comparing metadata tables (e.g., schema versions, >> state >>>>> entry >>>>>>>>>> counts) >>>>>>>>>>>> across catalogs reveals structural or quantitative >>>>> differences. >>>>>>>>>>>> 3. For deeper analysis, users could write SQL queries to >>>>> compare >>>>>>>>>> specific >>>>>>>>>>>> state partitions or leverage the metaservice to track >> state >>>>>>>> evolution >>>>>>>>>>>> (e.g., added/removed operators, modified state >>>>> configurations). >>>>>>>>>>>> >>>>>>>>>>>> If we plan to introduce a state catalog in the future, I >>>> would >>>>>>> lean >>>>>>>>>>> toward >>>>>>>>>>>> using metadata tables. If a utility tool can address the >>>>>>> challenges >>>>>>>>> we >>>>>>>>>>>> face, could we avoid introducing an additional connector? >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Shengkai >>>>>>>>>>>> >>>>>>>>>>>> Gyula Fóra <gyula.f...@gmail.com> 于2025年3月17日周一 20:25写道： >>>>>>>>>>>> >>>>>>>>>>>>> Hi All! >>>>>>>>>>>>> >>>>>>>>>>>>> Without going into too much detail here are my 2 cents >>>>>>> regarding >>>>>>>>> the >>>>>>>>>>>>> virtual column / catalog metadata / table (connector) >>>>>>> discussion >>>>>>>>> for >>>>>>>>>>> the >>>>>>>>>>>>> State metadata. >>>>>>>>>>>>> >>>>>>>>>>>>> State metadata such as the types of states, their >>>>> properties, >>>>>>>>> names, >>>>>>>>>>>> sizes >>>>>>>>>>>>> etc are all valuable information that can be used to >>>> enrich >>>>>>> the >>>>>>>>>>>>> computations we do on state. >>>>>>>>>>>>> We can either analyze it standalone (such as discover >>>>>>> anomalies, >>>>>>>>> for >>>>>>>>>>>> large >>>>>>>>>>>>> jobs with many states), across multiple savepoints >>>> (discover >>>>>>> how >>>>>>>>>> state >>>>>>>>>>>>> changed over time) or by joining it with keyed or >>>> non-keyed >>>>>>> state >>>>>>>>>> data >>>>>>>>>>> to >>>>>>>>>>>>> serve more complex queries on the state. >>>>>>>>>>>>> >>>>>>>>>>>>> The only solution that seems to serve all these >> use-cases >>>>> and >>>>>>>>>>>> requirements >>>>>>>>>>>>> in a straightforward and SQL canonical way is to simply >>>>> expose >>>>>>>> the >>>>>>>>>>> state >>>>>>>>>>>>> metadata as a separate table. This is a metadata table >>>> but >>>>> you >>>>>>>> can >>>>>>>>>> also >>>>>>>>>>>>> think of it as data table, it makes no practical >>>> difference >>>>>>> here. >>>>>>>>>>>>> >>>>>>>>>>>>> Once we have a catalog later, the catalog can offer >> this >>>>> table >>>>>>>> out >>>>>>>>> of >>>>>>>>>>> the >>>>>>>>>>>>> box, the same way databases provide metadata tables. >> For >>>>> this >>>>>>> to >>>>>>>>> work >>>>>>>>>>>>> however we need another, simpler connector that creates >>>> this >>>>>>>> table. >>>>>>>>>>>>> >>>>>>>>>>>>> +1 for state metadata as a separate connector/table, >>>> instead >>>>>>> of >>>>>>>>>> adding >>>>>>>>>>>>> virtual columns and adhoc catalog metadata that is hard >>>> to >>>>> use >>>>>>>> in a >>>>>>>>>>> large >>>>>>>>>>>>> number of queries. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Gyula >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 17, 2025 at 12:44 PM Gabor Somogyi < >>>>>>>>>>>> gabor.g.somo...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I’m planning on adding this, and we may collaborate >>>> on >>>>> it >>>>>>> in >>>>>>>>> the >>>>>>>>>>>>> future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> +1 on this, just ping me. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>>>>>>>>>>>>> >>>>>>>>>>>>>> After some code digging and POC all I can say that >> with >>>>>>> heavy >>>>>>>>>> effort >>>>>>>>>>> we >>>>>>>>>>>>> can >>>>>>>>>>>>>> maybe add such changes that we're able to show >> metadata >>>>> of a >>>>>>>>>>> savepoint >>>>>>>>>>>>> from >>>>>>>>>>>>>> catalog. >>>>>>>>>>>>>> I'm not against that but from user perspective this >> has >>>>>>> limited >>>>>>>>>>> value, >>>>>>>>>>>>> let >>>>>>>>>>>>>> me explain why. >>>>>>>>>>>>>> >>>>>>>>>>>>>> From high level perspective I see the following >> which I >>>>> see >>>>>>>>>> agreement >>>>>>>>>>>> on: >>>>>>>>>>>>>> * We should have a catalog which is representing one >> or >>>>> more >>>>>>>> jobs >>>>>>>>>>>>> savepoint >>>>>>>>>>>>>> data set (future plan) >>>>>>>>>>>>>> * Savepoints should be able to be registered in the >>>>> catalog >>>>>>>> which >>>>>>>>>> are >>>>>>>>>>>>> then >>>>>>>>>>>>>> databases (future plan) >>>>>>>>>>>>>> * There must be a possiblity to create tables from >>>>> databases >>>>>>>>> where >>>>>>>>>>>> users >>>>>>>>>>>>>> can read state data (exists already) >>>>>>>>>>>>>> >>>>>>>>>>>>>> In terms of metadata, If I understand correctly then >>>> the >>>>>>>>> suggested >>>>>>>>>>>>> approach >>>>>>>>>>>>>> would be to access >>>>>>>>>>>>>> it from the catalog describe command, right? Adding >>>> that >>>>>>> info >>>>>>>>> when >>>>>>>>>>>>> specific >>>>>>>>>>>>>> database describe command >>>>>>>>>>>>>> is executed could be done. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The question is for instance how can users create >> such >>>> a >>>>>>> logic >>>>>>>>> that >>>>>>>>>>>> tells >>>>>>>>>>>>>> them what is >>>>>>>>>>>>>> the difference between multiple savepoints? >>>>>>>>>>>>>> Just to give some examples: >>>>>>>>>>>>>> * per operator size changes between savepoints >>>>>>>>>>>>>> * show values from operator data where state size >>>> reaches >>>>> a >>>>>>>>>> boundary >>>>>>>>>>>>>> * in general "find which checkpoint ruined things" is >>>>> quite >>>>>>>>> common >>>>>>>>>>>>> pattern >>>>>>>>>>>>>> What I would like to highlight here is that from >> Flink >>>>>>> point of >>>>>>>>>> view >>>>>>>>>>>> the >>>>>>>>>>>>>> metadata can be >>>>>>>>>>>>>> considered as a static side output information but >> for >>>>> users >>>>>>>>> these >>>>>>>>>>>> values >>>>>>>>>>>>>> are actual real data >>>>>>>>>>>>>> where logic is planned to build around. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The metadata is more like one-time information >>>> instead >>>>> of >>>>>>> a >>>>>>>>>>> streaming >>>>>>>>>>>>>> data that changes all >>>>>>>>>>>>>> the time, so a single connector seems to be an >>>> overkill. >>>>>>>>>>>>>> >>>>>>>>>>>>>> State data is also static within a savepoint and >> that's >>>>> the >>>>>>>>> reason >>>>>>>>>>> why >>>>>>>>>>>>> the >>>>>>>>>>>>>> state processor API is working in batch mode. >>>>>>>>>>>>>> When we handle multiple checkpoints in a streaming >>>> fashion >>>>>>> then >>>>>>>>>> this >>>>>>>>>>>> can >>>>>>>>>>>>> be >>>>>>>>>>>>>> viewed from another angle. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We can come up with more lightweight solution other >>>> than a >>>>>>> new >>>>>>>>>>>> connector >>>>>>>>>>>>>> but enforcing users to parse the catalog >>>>>>>>>>>>>> describe command output in order to compare multiple >>>>>>> savepoints >>>>>>>>>>> doesn't >>>>>>>>>>>>>> sound smooth user experience. >>>>>>>>>>>>>> Honestly I've no other idea how exposing metadata as >>>> real >>>>>>> user >>>>>>>>> data >>>>>>>>>>> so >>>>>>>>>>>>>> waiting on other approaches. >>>>>>>>>>>>>> >>>>>>>>>>>>>> BR, >>>>>>>>>>>>>> G >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Mar 13, 2025 at 2:44 AM Shengkai Fang < >>>>>>>> fskm...@gmail.com >>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Looking forward to hearing the good news! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Shengkai >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Gabor Somogyi <gabor.g.somo...@gmail.com> >>>> 于2025年3月12日周三 >>>>>>>>> 22:24写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for both the valuable input! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Let me take a closer look at the suggestions, >> like >>>> the >>>>>>>>> Catalog >>>>>>>>>>>>>>> capabilities >>>>>>>>>>>>>>>> and possibility of embedding TypeInformation or >>>>>>>>>>>>>>>> StateDescriptor metadata directly into the raw >>>> state >>>>>>>> files... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>> G >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 8:17 AM Shengkai Fang < >>>>>>>>>> fskm...@gmail.com >>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for Zakelly's clarification. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 to delay the discussion about this. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I’d like to share my perspective on the State >>>>> Catalog >>>>>>>>>> proposal. >>>>>>>>>>>>> While >>>>>>>>>>>>>>>>> introducing this capability is beneficial, >> there >>>> is >>>>> a >>>>>>>>>> blocker: >>>>>>>>>>>> the >>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>> StateBackend architecture does not permit >>>> operators >>>>> to >>>>>>>>> encode >>>>>>>>>>>>>>>>> TypeInformation into the state—it only >> preserves >>>> the >>>>>>>>>>> Serializer. >>>>>>>>>>>>> This >>>>>>>>>>>>>>>>> limitation creates an asymmetry, as operators >>>> alone >>>>>>>> retain >>>>>>>>>>>>> knowledge >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> data structure’s schema. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To address this, I suggest allowing operators >> to >>>>> embed >>>>>>>>>>>>>> TypeInformation >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>> StateDescriptor metadata directly into the raw >>>> state >>>>>>>> files. >>>>>>>>>>> Such >>>>>>>>>>>> a >>>>>>>>>>>>>>> design >>>>>>>>>>>>>>>>> would enable the Catalog to: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Parse state files and programmatically >> derive >>>> the >>>>>>>> schema >>>>>>>>>> and >>>>>>>>>>>>>>>> structural >>>>>>>>>>>>>>>>> guarantees for each state. >>>>>>>>>>>>>>>>> 2. Leverage existing Flink Table utilities, >> such >>>> as >>>>>>>>>>>>>>>>> LegacyTypeInfoDataTypeConverter (in >>>>>>>>>>>>>>> org.apache.flink.table.types.utils), >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> bridge TypeInformation and DataType >> conversions. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If we can not store the TypeInformation or >>>>>>>> StateDescriptor >>>>>>>>>> into >>>>>>>>>>>> the >>>>>>>>>>>>>> raw >>>>>>>>>>>>>>>>> state files, I am +1 for this FLIP to use >>>> metadata >>>>>>> column >>>>>>>>> to >>>>>>>>>>>>> retrieve >>>>>>>>>>>>>>>>> information. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Shengkai >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Zakelly Lan <zakelly....@gmail.com> >>>> 于2025年3月12日周三 >>>>>>>> 12:43写道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Gabor and Shengkai, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for sharing your thoughts! This is a >>>> long >>>>>>>>> discussion >>>>>>>>>>> and >>>>>>>>>>>>>> sorry >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> the late reply (I'm busy catching up with >>>> release >>>>>>> 2.0 >>>>>>>>> these >>>>>>>>>>>>> days). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Let me first clarify your thoughts to ensure >> I >>>>>>>> understand >>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>>> IIUC, >>>>>>>>>>>>>>>>>> there is no persistent configuration for >> state >>>> TTL >>>>>>> in >>>>>>>> the >>>>>>>>>>>>>> checkpoint. >>>>>>>>>>>>>>>>> While >>>>>>>>>>>>>>>>>> you can infer that TTL is enabled by reading >>>> the >>>>>>>>>> serializer, >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> checkpoint >>>>>>>>>>>>>>>>>> itself only stores the last access time for >>>> each >>>>>>> value. >>>>>>>>> So >>>>>>>>>>> the >>>>>>>>>>>>> only >>>>>>>>>>>>>>>> thing >>>>>>>>>>>>>>>>>> we can show is the last access time for each >>>>> value. >>>>>>> But >>>>>>>>> it >>>>>>>>>> is >>>>>>>>>>>> not >>>>>>>>>>>>>>>>> required >>>>>>>>>>>>>>>>>> for all state backends to store this, as they >>>> may >>>>>>>>> directly >>>>>>>>>>>> store >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> expired time. This will also increase the >>>>>>> difficulty of >>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>> & >>>>>>>>>>>>>>>>>> maintenance. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This once again reiterates the importance of >>>>> unified >>>>>>>>>> metadata >>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> checkpoints. I’m planning on adding this, and >>>> we >>>>> may >>>>>>>>>>>> collaborate >>>>>>>>>>>>> on >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the future. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata Column >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm not in favor of adding a new connector >> for >>>>>>>> metadata. >>>>>>>>>> The >>>>>>>>>>>>>> metadata >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>> more like one-time information instead of a >>>>>>> streaming >>>>>>>>> data >>>>>>>>>>> that >>>>>>>>>>>>>>> changes >>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>> the time, so a single connector seems to be >> an >>>>>>>> overkill. >>>>>>>>> It >>>>>>>>>>> is >>>>>>>>>>>>> not >>>>>>>>>>>>>>> easy >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> withdraw a connector if we have a better >>>> solution >>>>> in >>>>>>>>>> future. >>>>>>>>>>>> I'm >>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>> familiar with current Catalog capabilities, >>>> and if >>>>>>> it >>>>>>>>> could >>>>>>>>>>>>> extract >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> show some operator-level information from >>>>> savepoint, >>>>>>>> that >>>>>>>>>>> would >>>>>>>>>>>>> be >>>>>>>>>>>>>>>> great. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If the Catalog can't do that, I would >> consider >>>> the >>>>>>>>> current >>>>>>>>>>> FLIP >>>>>>>>>>>>> to >>>>>>>>>>>>>>> be a >>>>>>>>>>>>>>>>>> compromise solution. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> And if we have that unified metadata for >>>>>>>>>> checkpoint/savepoint >>>>>>>>>>>> in >>>>>>>>>>>>>>>> future, >>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>> may directly register savepoint in catalog, >> and >>>>>>> create >>>>>>>> a >>>>>>>>>>> source >>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>> specifying complex columns, as well as >> describe >>>>> the >>>>>>>>>> savepoint >>>>>>>>>>>>>> catalog >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> get the metadata. That's a good solution in >> my >>>>> mind. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> Zakelly >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 10:35 AM Shengkai >> Fang >>>> < >>>>>>>>>>>>> fskm...@gmail.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Gabor, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>>>>>> `savepoint-metadata` >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I would argue against introducing a new >>>>> connector >>>>>>>> type >>>>>>>>>>> named >>>>>>>>>>>>>>>>>>> savepoint-metadata, as the existing Catalog >>>>>>> mechanism >>>>>>>>> can >>>>>>>>>>>>>>> inherently >>>>>>>>>>>>>>>>>>> provide the necessary connector factory >>>>>>> capabilities. >>>>>>>>>> I’ve >>>>>>>>>>>>>> detailed >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> proposal in branch[1]. Please take a moment >>>> to >>>>>>> review >>>>>>>>> it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If we introduce a connector named >>>>>>>> `savepoint-metadata`, >>>>>>>>>> it >>>>>>>>>>>>> means >>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> create a temporary table with connector >>>>>>>>>>> `savepoint-metadata` >>>>>>>>>>>>> and >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> connector needs to check whether table >>>> schema is >>>>>>> same >>>>>>>>> to >>>>>>>>>>> the >>>>>>>>>>>>>> schema >>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>> proposed in the FLIP. On the other hand, >> it's >>>>> not >>>>>>>> easy >>>>>>>>>> work >>>>>>>>>>>> for >>>>>>>>>>>>>>>> others >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> users a metadata table with same schema. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://github.com/apache/flink/compare/master...fsk119:flink:state-metadata?expand=1#diff-712a7bc92fe46c405fb0e61b475bb2a005cb7a72bab7df28bbb92744bcb5f465R63 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>> Shengkai >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Gabor Somogyi <gabor.g.somo...@gmail.com> >>>>>>>>> 于2025年3月11日周二 >>>>>>>>>>>>> 16:56写道： >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Shengkai, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> From directional perspective I agree your >>>> idea >>>>>>> how >>>>>>>> it >>>>>>>>>> can >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>> implemented. >>>>>>>>>>>>>>>>>>>> Previously I've mentioned that TTL >>>> information >>>>>>> is >>>>>>>> not >>>>>>>>>>>> exposed >>>>>>>>>>>>>> on >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>> processor API (which the SQL state >>>> connector >>>>>>> uses >>>>>>>> to >>>>>>>>>> read >>>>>>>>>>>>> data) >>>>>>>>>>>>>>>>>>>> and unless somebody show me the opposite >>>> this >>>>>>> FLIP >>>>>>>> is >>>>>>>>>> not >>>>>>>>>>>>> going >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> address >>>>>>>>>>>>>>>>>>>> this to avoid feature creep. Our users >> are >>>>> also >>>>>>>>>>> interested >>>>>>>>>>>> in >>>>>>>>>>>>>> TTL >>>>>>>>>>>>>>>> so >>>>>>>>>>>>>>>>>>>> sooner or later we're going to expose it, >>>> this >>>>>>> is >>>>>>>>>> matter >>>>>>>>>>> of >>>>>>>>>>>>>>>>> scheduling. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>>>>>>> `savepoint-metadata` >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Not sure I understand your point at all >>>>> related >>>>>>>>>>>> StateCatalog. >>>>>>>>>>>>>>> First >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>> I can't agree more that StateCatalog is >>>> needed >>>>>>> and >>>>>>>>> is a >>>>>>>>>>>>> planned >>>>>>>>>>>>>>>>>> building >>>>>>>>>>>>>>>>>>>> block in an upcoming >>>>>>>>>>>>>>>>>>>> FLIP but not sure how can it help now? No >>>>> matter >>>>>>>>> what, >>>>>>>>>>> your >>>>>>>>>>>>>>>> knowledge >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>> essential when we add StateCatalog. Let >> me >>>>>>> expose >>>>>>>> my >>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> area: >>>>>>>>>>>>>>>>>>>> * First we need create table statements >> to >>>>>>> access >>>>>>>>> state >>>>>>>>>>>> data >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> metadata >>>>>>>>>>>>>>>>>>>> * When we have that then we can add >>>>> StateCatalog >>>>>>>>> which >>>>>>>>>>>> could >>>>>>>>>>>>>>>>>> potentially >>>>>>>>>>>>>>>>>>>> ease the life of users by for ex. giving >>>>>>>>> off-the-shelf >>>>>>>>>>>> tables >>>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>>>> sweating with create table statements >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> User expectations: >>>>>>>>>>>>>>>>>>>> * See state data (this is fulfilled with >>>> the >>>>>>>> existing >>>>>>>>>>>>>> connector) >>>>>>>>>>>>>>>>>>>> * See metadata about state data like TTL >>>> (this >>>>>>> can >>>>>>>> be >>>>>>>>>>> added >>>>>>>>>>>>> as >>>>>>>>>>>>>>>>> metadata >>>>>>>>>>>>>>>>>>>> column as you suggested since it belongs >> to >>>>> the >>>>>>>> data) >>>>>>>>>>>>>>>>>>>> * See metadata about operators (this can >> be >>>>>>> added >>>>>>>>> from >>>>>>>>>>>>>>>>>>> savepoint-metadata) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Important to highlight that state data >>>> table >>>>>>> format >>>>>>>>>>> differs >>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>> metadata table format. Namely one table >> has >>>>> rows >>>>>>>> for >>>>>>>>>>> state >>>>>>>>>>>>>> values >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> another has rows for operators, right? >>>>>>>>>>>>>>>>>>>> I think that's the reason why you've >>>>> pinpointed >>>>>>> out >>>>>>>>>> that >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> suggested >>>>>>>>>>>>>>>>>>>> metadata columns are somewhat clunky. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> As a conclusion I agree to add >>>>> ${state-name}_ttl >>>>>>>>>> metadata >>>>>>>>>>>>>> column >>>>>>>>>>>>>>>>> later >>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>> since it belongs to the state value and >>>>> adding a >>>>>>>> new >>>>>>>>>>> table >>>>>>>>>>>>> type >>>>>>>>>>>>>>>> (like >>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>> suggested similar to PG [1]) >>>>>>>>>>>>>>>>>>>> for metadata. Please see how Spark does >>>> that >>>>> too >>>>>>>> [2]. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> If you have better approach then please >>>>>>> elaborate >>>>>>>>> with >>>>>>>>>>> more >>>>>>>>>>>>>>> details >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> help me to understand your point. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>>>> savepoints >>>>>>>> that >>>>>>>>>> the >>>>>>>>>>>>> number >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> keys >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key >>>> state >>>>>>>> itself. >>>>>>>>>>>>>>>>>>>>> But again, this is a good feature as-is >>>> and >>>>>>> can >>>>>>>> be >>>>>>>>>>>> handled >>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>>> separate >>>>>>>>>>>>>>>>>>>>> jira. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I've just created >>>>>>>>>>>>>>>> >> https://issues.apache.org/jira/browse/FLINK-37456. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> >>>>> https://www.postgresql.org/docs/current/view-pg-tables.html >>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>> G >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, Mar 11, 2025 at 3:55 AM Shengkai >>>> Fang >>>>> < >>>>>>>>>>>>>> fskm...@gmail.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your response. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thank you for addressing the >> limitations >>>>> here. >>>>>>>>>>> However, I >>>>>>>>>>>>>>> believe >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> be beneficial to further clarify the >> API >>>> in >>>>>>> this >>>>>>>>> FLIP >>>>>>>>>>>>>> regarding >>>>>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>> can specify the TTL column. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> One potential approach that comes to >>>> mind is >>>>>>>> using >>>>>>>>> a >>>>>>>>>>>>>>> standardized >>>>>>>>>>>>>>>>>>> naming >>>>>>>>>>>>>>>>>>>>> convention such as ${state-name}_ttl >> for >>>> the >>>>>>>>> metadata >>>>>>>>>>>>> column >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> defines >>>>>>>>>>>>>>>>>>>>> the TTL value. In terms of >>>> implementation, >>>>> the >>>>>>>>>>>>>>>> listReadableMetadata >>>>>>>>>>>>>>>>>>>>> function could: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. Read the table’s columns and >>>>> configuration, >>>>>>>>>>>>>>>>>>>>> 2. Extract all defined state names, and >>>>>>>>>>>>>>>>>>>>> 3. Return a structured list of metadata >>>>>>> entries >>>>>>>>>>> formatted >>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>> ${state-name}_ttl. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. Adding a new connector with >>>>>>>>> `savepoint-metadata` >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Introducing a new connector type at >> this >>>>> stage >>>>>>>> may >>>>>>>>>>>>>>> unnecessarily >>>>>>>>>>>>>>>>>>>> complicate >>>>>>>>>>>>>>>>>>>>> the system. Given that every table >>>> already >>>>>>>> belongs >>>>>>>>>> to a >>>>>>>>>>>>>>> Catalog, >>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> designed to provide a Factory for >>>> building >>>>>>> source >>>>>>>>> or >>>>>>>>>>> sink >>>>>>>>>>>>>>>>>> connectors, I >>>>>>>>>>>>>>>>>>>>> propose integrating a dedicated >>>> StateCatalog >>>>>>>>> instead. >>>>>>>>>>>> This >>>>>>>>>>>>>>>> approach >>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> allow us to: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. Leverage the Catalog’s existing >>>>>>> capabilities >>>>>>>> to >>>>>>>>>>> manage >>>>>>>>>>>>> TTL >>>>>>>>>>>>>>>>>> metadata >>>>>>>>>>>>>>>>>>>>> (e.g., state names and TTL logic) >> without >>>>>>>>> duplicating >>>>>>>>>>>>>>>>> functionality. >>>>>>>>>>>>>>>>>>>>> 2. Provide a unified interface for >>>> connector >>>>>>>>>>>> instantiation >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> metadata >>>>>>>>>>>>>>>>>>>>> handling through the Catalog’s Factory >>>>>>> pattern. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Would this design decision better align >>>> with >>>>>>> our >>>>>>>>>>>>>> architecture’s >>>>>>>>>>>>>>>>>>>>> extensibility and reduce redundancy? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>>>>>> savepoints >>>>>>>>> that >>>>>>>>>>> the >>>>>>>>>>>>>> number >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> keys >>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key >>>>> state >>>>>>>>> itself. >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature >> as-is >>>>> and >>>>>>> can >>>>>>>>> be >>>>>>>>>>>>> handled >>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>>>> separate >>>>>>>>>>>>>>>>>>>>>> jira. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> +1 for a separate jira. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>> Shengkai >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < >> gabor.g.somo...@gmail.com >>>>> >>>>>>>>>>> 于2025年3月10日周一 >>>>>>>>>>>>>>> 19:05写道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Shengkai, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Please see my comments inline. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>> G >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 3, 2025 at 7:07 AM >> Shengkai >>>>>>> Fang < >>>>>>>>>>>>>>>> fskm...@gmail.com> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi, Gabor. Thanks for your the >> FLIP. >>>> I >>>>>>> have >>>>>>>>> some >>>>>>>>>>>>>> questions >>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> FLIP: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 1. State TTL for Value Columns >>>>>>>>>>>>>>>>>>>>>>> How can users retrieve the state >> TTL >>>>>>>>>> (Time-to-Live) >>>>>>>>>>>> for >>>>>>>>>>>>>>> each >>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>> column? >>>>>>>>>>>>>>>>>>>>>>> From my understanding of the >> current >>>>>>> design, >>>>>>>> it >>>>>>>>>>> seems >>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>> functionality is not supported. >> Could >>>>> you >>>>>>>>> clarify >>>>>>>>>>> if >>>>>>>>>>>>>> there >>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>> plans >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> address this limitation? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Since the state processor API is not >>>> yet >>>>>>>> exposing >>>>>>>>>>> this >>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>> would require several steps. >>>>>>>>>>>>>>>>>>>>>> First, the state processor API >> support >>>>>>> needs to >>>>>>>>> be >>>>>>>>>>>> added >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>> exposed on the SQL API. >>>>>>>>>>>>>>>>>>>>>> This is definitely a future >> improvement >>>>>>> which >>>>>>>> is >>>>>>>>>>> useful >>>>>>>>>>>>> and >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> handled >>>>>>>>>>>>>>>>>>>>>> in a separate jira. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 2. Metadata Table vs. Metadata >> Column >>>>>>>>>>>>>>>>>>>>>>> The metadata information described >> in >>>>> the >>>>>>>> FLIP >>>>>>>>>>>> appears >>>>>>>>>>>>> to >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>> intended >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> describe the state files stored at >> a >>>>>>> specific >>>>>>>>>>>> location. >>>>>>>>>>>>>> To >>>>>>>>>>>>>>>> me, >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>> concept >>>>>>>>>>>>>>>>>>>>>>> aligns more closely with system >>>> tables >>>>>>> like >>>>>>>>>>> pg_tables >>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> PostgreSQL >>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>> the INFORMATION_SCHEMA in MySQL >> [2]. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Adding a new connector with >>>>>>>> `savepoint-metadata` >>>>>>>>>> is a >>>>>>>>>>>>>>>> possibility >>>>>>>>>>>>>>>>>>> where >>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>> can create such functionality. >>>>>>>>>>>>>>>>>>>>>> I'm not against that, just want to >>>> have a >>>>>>>> common >>>>>>>>>>>>> agreement >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>> like to move that direction. >>>>>>>>>>>>>>>>>>>>>> (As a side note not just PG but Spark >>>> also >>>>>>> has >>>>>>>>>>> similar >>>>>>>>>>>>>>> approach >>>>>>>>>>>>>>>>>> and I >>>>>>>>>>>>>>>>>>>>>> basically like the idea). >>>>>>>>>>>>>>>>>>>>>> If we would go that direction >> savepoint >>>>>>>> metadata >>>>>>>>>> can >>>>>>>>>>> be >>>>>>>>>>>>>>> reached >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>>>>>>>> that one row would represent >>>>>>>>>>>>>>>>>>>>>> an operator with it's values >> something >>>>> like >>>>>>>> this: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┐ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> │operatorN│operatorU│operatorH│paralleli│maxParall│subtaskSt│coordinat│totalSta│ >>>>>>>>>>>>>>>>>>>>>> │ame │id │ash │sm >>>>>>> │elism >>>>>>>>>>>>>>>>>>>>>> │atesCount│orStateSi│tesSizeI│ >>>>>>>>>>>>>>>>>>>>>> │ │ │ │ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │zeInBytes│nBytes │ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>>>>>>>>>>>>>>>>>>>>> │Source: │datagen-s│47aee9439│2 >>>>> │128 >>>>>>>>>> │2 >>>>>>>>>>>>>>> │16 >>>>>>>>>>>>>>>>>>>>>> │546 │ >>>>>>>>>>>>>>>>>>>>>> │datagen-s│ource-uid│4d6ea26e2│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ource │ │d544bef0a│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ │ │37bb5 │ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>>>>>>>>>>>>>>>>>>>>> │long-udf-│long-udf-│6ed3f40bf│2 >>>>> │128 >>>>>>>>>> │2 >>>>>>>>>>>>>>> │0 >>>>>>>>>>>>>>>>>>>> │0 >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │with-mast│with-mast│f3c8dfcdf│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │er-hook │er-hook-u│cb95128a1│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ │id │018f1 │ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>>>>>>>>>>>>>>>>>>>>> │value-pro│value-pro│ca4f5fe9a│2 >>>>> │128 >>>>>>>>>> │2 >>>>>>>>>>>>>>> │0 >>>>>>>>>>>>>>>>>>>>>> │40726 │ >>>>>>>>>>>>>>>>>>>>>> │cess │cess-uid │637b656f0│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ │ │9ea78b3e7│ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ │ │a15b9 │ >>>> │ >>>>>>>>> │ >>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> │ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> ├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> This table can then be joined with >> the >>>>>>> actually >>>>>>>>>>>> existing >>>>>>>>>>>>>>>>>> `savepoint` >>>>>>>>>>>>>>>>>>>>>> connector created tables based on UID >>>> hash >>>>>>>> (which >>>>>>>>>> is >>>>>>>>>>>>> unique >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>>>> exists). >>>>>>>>>>>>>>>>>>>>>> This would mean that the already >>>> existing >>>>>>> table >>>>>>>>>> would >>>>>>>>>>>>> need >>>>>>>>>>>>>>>> only a >>>>>>>>>>>>>>>>>>>> single >>>>>>>>>>>>>>>>>>>>>> metadata column which is the UID >> hash. >>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>> @zakelly, plz share your thoughts >> too. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> If we opt to use metadata columns, >>>> every >>>>>>>> record >>>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>> up >>>>>>>>>>>>>>>>>>>>>>> having identical values for these >>>>> columns >>>>>>>>> (please >>>>>>>>>>>>> correct >>>>>>>>>>>>>>> me >>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>> I’m >>>>>>>>>>>>>>>>>>>>>>> mistaken). On the other hand, the >>>> state >>>>>>>>> connector >>>>>>>>>>>>>> requires >>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>>>>>>>>>> an operator UID or operator UID >> hash, >>>>>>> after >>>>>>>>> which >>>>>>>>>>> it >>>>>>>>>>>>>>> outputs >>>>>>>>>>>>>>>>>>>>> user-defined >>>>>>>>>>>>>>>>>>>>>>> values in its records. This >> approach >>>>> feels >>>>>>>>>> somewhat >>>>>>>>>>>>>>> redundant >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> me. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If we would add a new >>>> `savepoint-metadata` >>>>>>>>>> connector >>>>>>>>>>>> then >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> addressed. >>>>>>>>>>>>>>>>>>>>>> On the other hand UID and UID hash >> are >>>>>>> having >>>>>>>>>>> either-or >>>>>>>>>>>>>>>>>> relationship >>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>> config perspective, >>>>>>>>>>>>>>>>>>>>>> so when a user provides the UID then >>>>> he/she >>>>>>> can >>>>>>>>> be >>>>>>>>>>>>>> interested >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> hash >>>>>>>>>>>>>>>>>>>>>> for further calculations >>>>>>>>>>>>>>>>>>>>>> (the whole Flink internals are >>>> depending >>>>> on >>>>>>> the >>>>>>>>>>> hash). >>>>>>>>>>>>>>> Printing >>>>>>>>>>>>>>>>> out >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> human readable UID >>>>>>>>>>>>>>>>>>>>>> is an explicit requirement from the >>>> user >>>>>>> side >>>>>>>>>> because >>>>>>>>>>>>>> hashes >>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> human >>>>>>>>>>>>>>>>>>>>>> readable. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 3. Handling LIST and MAP States in >>>> the >>>>>>> State >>>>>>>>>>>> Connector >>>>>>>>>>>>>>>>>>>>>>> I have concerns about how the >> current >>>>>>> design >>>>>>>>>>> handles >>>>>>>>>>>>> LIST >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> MAP >>>>>>>>>>>>>>>>>>>>> states. >>>>>>>>>>>>>>>>>>>>>>> Specifically, the state connector >>>> uses >>>>>>> Flink >>>>>>>>>> SQL’s >>>>>>>>>>>> MAP >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> ARRAY >>>>>>>>>>>>>>>>>>>> types, >>>>>>>>>>>>>>>>>>>>>>> which implies that it attempts to >>>> load >>>>>>> entire >>>>>>>>> MAP >>>>>>>>>>> or >>>>>>>>>>>>> LIST >>>>>>>>>>>>>>>>> states >>>>>>>>>>>>>>>>>>> into >>>>>>>>>>>>>>>>>>>>>>> memory. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> However, in many real-world >>>> scenarios, >>>>>>> these >>>>>>>>>> states >>>>>>>>>>>> can >>>>>>>>>>>>>>> grow >>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>>>>>> large. >>>>>>>>>>>>>>>>>>>>>>> Typically, the state API addresses >>>> this >>>>> by >>>>>>>>>>> providing >>>>>>>>>>>> an >>>>>>>>>>>>>>>>> iterator >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> traverse elements within the state >>>>>>>>> incrementally. >>>>>>>>>>> I’m >>>>>>>>>>>>>>> unsure >>>>>>>>>>>>>>>>>>> whether >>>>>>>>>>>>>>>>>>>>> I’ve >>>>>>>>>>>>>>>>>>>>>>> missed something in FLIP-496 or >>>>> FLIP-512, >>>>>>> but >>>>>>>>> it >>>>>>>>>>>> seems >>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>>>>>>>> design might struggle with >>>> scalability >>>>> in >>>>>>>> such >>>>>>>>>>> cases. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> You see it good, the current >>>>> implementation >>>>>>>> keeps >>>>>>>>>>> state >>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>> single >>>>>>>>>>>>>>>>>>>> key >>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> memory. >>>>>>>>>>>>>>>>>>>>>> Back in the days we've considered >> this >>>>>>>> potential >>>>>>>>>>> issue >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> concluded >>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>> this is not necessarily >>>>>>>>>>>>>>>>>>>>>> needed for the initial version and >> can >>>> be >>>>>>> done >>>>>>>>> as a >>>>>>>>>>>> later >>>>>>>>>>>>>>>>>>> improvement. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Up until now we've seen even in TB >>>>>>> savepoints >>>>>>>>> that >>>>>>>>>>> the >>>>>>>>>>>>>> number >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> keys >>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>> be extremely huge but not the per key >>>>> state >>>>>>>>> itself. >>>>>>>>>>>>>>>>>>>>>> But again, this is a good feature >> as-is >>>>> and >>>>>>> can >>>>>>>>> be >>>>>>>>>>>>> handled >>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>>>> separate >>>>>>>>>>>>>>>>>>>>>> jira. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>> Shengkai >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>> >>>>>>>>> https://www.postgresql.org/docs/current/view-pg-tables.html >>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Gabor Somogyi < >>>>> gabor.g.somo...@gmail.com> >>>>>>>>>>>> 于2025年3月3日周一 >>>>>>>>>>>>>>>>> 02:00写道： >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Zakelly, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> In order to shoot for simplicity >>>>>>> `METADATA >>>>>>>>>>> VIRTUAL` >>>>>>>>>>>>> as >>>>>>>>>>>>>>> key >>>>>>>>>>>>>>>>>> words >>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>> definition is the target. >>>>>>>>>>>>>>>>>>>>>>>> When it's not super complex the >>>> latter >>>>>>> can >>>>>>>> be >>>>>>>>>>> added >>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>> G >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 2, 2025 at 3:37 PM >>>> Zakelly >>>>>>> Lan >>>>>>>> < >>>>>>>>>>>>>>>>>>> zakelly....@gmail.com> >>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Gabor, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> +1 for this. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Will the metadata column use >>>>> `METADATA >>>>>>>>>> VIRTUAL` >>>>>>>>>>>> as >>>>>>>>>>>>>> key >>>>>>>>>>>>>>>>> words >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> definition, or `METADATA FROM >> xxx >>>>>>>> VIRTUAL` >>>>>>>>>> for >>>>>>>>>>>>>>> renaming, >>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> Kafka table? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>> Zakelly >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Mar 1, 2025 at 1:31 PM >>>> Gabor >>>>>>>>> Somogyi >>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>> gabor.g.somo...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start a >> discussion >>>> of >>>>>>>>> FLIP-512: >>>>>>>>>>> Add >>>>>>>>>>>>>> meta >>>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> SQL >>>>>>>>>>>>>>>>>>>>>>>>>> state connector [1]. >>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to add your >> thoughts >>>> to >>>>>>> make >>>>>>>>> this >>>>>>>>>>>>> feature >>>>>>>>>>>>>>>>> better. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>>>> G >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
