Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP: 1. State TTL for Value Columns How can users retrieve the state TTL (Time-to-Live) for each value column? >From my understanding of the current design, it seems that this functionality is not supported. Could you clarify if there are plans to address this limitation?
2. Metadata Table vs. Metadata Column The metadata information described in the FLIP appears to be intended to describe the state files stored at a specific location. To me, this concept aligns more closely with system tables like pg_tables in PostgreSQL [1] or the INFORMATION_SCHEMA in MySQL [2]. If we opt to use metadata columns, every record in the table would end up having identical values for these columns (please correct me if I’m mistaken). On the other hand, the state connector requires users to specify an operator UID or operator UID hash, after which it outputs user-defined values in its records. This approach feels somewhat redundant to me. 3. Handling LIST and MAP States in the State Connector I have concerns about how the current design handles LIST and MAP states. Specifically, the state connector uses Flink SQL’s MAP and ARRAY types, which implies that it attempts to load entire MAP or LIST states into memory. However, in many real-world scenarios, these states can grow very large. Typically, the state API addresses this by providing an iterator to traverse elements within the state incrementally. I’m unsure whether I’ve missed something in FLIP-496 or FLIP-512, but it seems that the current design might struggle with scalability in such cases. Best, Shengkai [1] https://www.postgresql.org/docs/current/view-pg-tables.html [2] https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月3日周一 02:00写道： > Hi Zakelly, > > In order to shoot for simplicity `METADATA VIRTUAL` as key words for > definition is the target. > When it's not super complex the latter can be added too. > > BR, > G > > > On Sun, Mar 2, 2025 at 3:37 PM Zakelly Lan <zakelly....@gmail.com> wrote: > > > Hi Gabor, > > > > +1 for this. > > > > Will the metadata column use `METADATA VIRTUAL` as key words for > > definition, or `METADATA FROM xxx VIRTUAL` for renaming, just like the > > Kafka table? > > > > > > Best, > > Zakelly > > > > On Sat, Mar 1, 2025 at 1:31 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > I'd like to start a discussion of FLIP-512: Add meta information to SQL > > > state connector [1]. > > > Feel free to add your thoughts to make this feature better. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector > > > > > > BR, > > > G > > > > > >
