Hi Shengkai, Thanks for the questions. This week I'm off but will answer them when I'm back.
G On Mon, Mar 3, 2025, 07:07 Shengkai Fang <fskm...@gmail.com> wrote: > Hi, Gabor. Thanks for your the FLIP. I have some questions about the FLIP: > > 1. State TTL for Value Columns > How can users retrieve the state TTL (Time-to-Live) for each value column? > From my understanding of the current design, it seems that this > functionality is not supported. Could you clarify if there are plans to > address this limitation? > > 2. Metadata Table vs. Metadata Column > The metadata information described in the FLIP appears to be intended to > describe the state files stored at a specific location. To me, this concept > aligns more closely with system tables like pg_tables in PostgreSQL [1] or > the INFORMATION_SCHEMA in MySQL [2]. > > If we opt to use metadata columns, every record in the table would end up > having identical values for these columns (please correct me if I’m > mistaken). On the other hand, the state connector requires users to specify > an operator UID or operator UID hash, after which it outputs user-defined > values in its records. This approach feels somewhat redundant to me. > > 3. Handling LIST and MAP States in the State Connector > I have concerns about how the current design handles LIST and MAP states. > Specifically, the state connector uses Flink SQL’s MAP and ARRAY types, > which implies that it attempts to load entire MAP or LIST states into > memory. > > However, in many real-world scenarios, these states can grow very large. > Typically, the state API addresses this by providing an iterator to > traverse elements within the state incrementally. I’m unsure whether I’ve > missed something in FLIP-496 or FLIP-512, but it seems that the current > design might struggle with scalability in such cases. > > Best, > Shengkai > > [1] https://www.postgresql.org/docs/current/view-pg-tables.html > [2] > > https://dev.mysql.com/doc/refman/8.4/en/information-schema-tables-table.html > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2025年3月3日周一 02:00写道： > > > Hi Zakelly, > > > > In order to shoot for simplicity `METADATA VIRTUAL` as key words for > > definition is the target. > > When it's not super complex the latter can be added too. > > > > BR, > > G > > > > > > On Sun, Mar 2, 2025 at 3:37 PM Zakelly Lan <zakelly....@gmail.com> > wrote: > > > > > Hi Gabor, > > > > > > +1 for this. > > > > > > Will the metadata column use `METADATA VIRTUAL` as key words for > > > definition, or `METADATA FROM xxx VIRTUAL` for renaming, just like the > > > Kafka table? > > > > > > > > > Best, > > > Zakelly > > > > > > On Sat, Mar 1, 2025 at 1:31 PM Gabor Somogyi < > gabor.g.somo...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > I'd like to start a discussion of FLIP-512: Add meta information to > SQL > > > > state connector [1]. > > > > Feel free to add your thoughts to make this feature better. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-512%3A+Add+meta+information+to+SQL+state+connector > > > > > > > > BR, > > > > G > > > > > > > > > >
