Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

If I understand correctly you use purely SQL workloads for savepoint creation. In short such case the initial version won't be much help. In a more detailed version SQL operators are creating non-keyed/operator state data which is going to be supported later on. The first version supports user cod

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Gabor, - Expose state metadata on data stream API > - Add metadata support for the SQL connector > - Add non-keyed/operator state data support for the SQL connector > - Add state catalog Thanks for this FLIP. This will be a great addition. Although, most of our use cases are SQL and not DataS

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Thanks for your clarification. Looking forward to the following FLIPs. Best, Shengkai Gabor Somogyi 于2024年12月23日周一 18:32写道： > Hi Shengkai, > > Thanks for your comments. > > Both state catalog and metadata tables are on the roadmap but not in this > FLIP. > The main approach is to do things incr

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Shengkai, Thanks for your comments. Both state catalog and metadata tables are on the roadmap but not in this FLIP. The main approach is to do things incrementally in order to avoid infinite discussion threads and 10k+ lines pull requests. Just to give some teaser, this is how I can roughly i

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi. Thanks Gabor's great FLIP. It is very useful for many cases. But I have some suggestions about this FLIP: 1. It's better to introduce a state Catalog that helps users to translate the state information to a table schema. The state catalog uses ``.`` to find required state and uses state descr

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Yanquan, Thanks for the question, please see the FLIP content related this. ---BEGIN FLIP CONTENT--- The target of this implementation proposal is to provide SQL support for keyed states. Other types are going to be covered in further FLIP documents. ---END FLIP CONTENT--- Keyed state data wh

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi, Gabor. Thanks for driving this FLIP and this immediately reminded me of exploring the status of the Source connector like Kafka or MySQLCDC to understand the progress of reading information. In SourceOperator, the state of reader in contained in a ListState named `SourceReaderState`[1], how

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Agreed, making the adjustments in the FLIP... On Thu, Dec 19, 2024 at 11:33 AM Zakelly Lan wrote: > Hi Gabor, > > Yes, that's what I meant. > > > Best, > Zakelly > > On Thu, Dec 19, 2024 at 6:22 PM Gabor Somogyi > wrote: > > > Hi Zakelly, > > > > Just for my own clear understanding the suggesti

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Gabor, Yes, that's what I meant. Best, Zakelly On Thu, Dec 19, 2024 at 6:22 PM Gabor Somogyi wrote: > Hi Zakelly, > > Just for my own clear understanding the suggestion is the following? > > state.type=keyed|non-keyed|operator|metadata > > Where non-keyed == operator? > I'm fine with this.

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Zakelly, Just for my own clear understanding the suggestion is the following? state.type=keyed|non-keyed|operator|metadata Where non-keyed == operator? I'm fine with this. BR, G On Thu, Dec 19, 2024 at 10:31 AM Zakelly Lan wrote: > Hi Gabor, > > The proposed configs look good to me. I al

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Gabor, The proposed configs look good to me. I also suggest that 'operator' can serve as an equivalent to 'non-keyed', as non-keyed state is also called operator state [1]. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#operator-state Best

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Zakelly, Fair point to have short names. How about having the following configs then? connector=savepoint state.type=keyed|non-keyed|metadata This would introduce the proper keyed/non-keyed option + the config pair will exactly tell that savepoint metadata is going to be read. BR, G On Thu

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Gabor, Thanks for your answers! I'm good with naming it 'savepoint' with an extra option like 'mode' specifying the reading type, or suffixes in names like 'savepoint-metadata' for each type of state. For the second option it means the current connector will be named 'savepoint-keyed'? Well I

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Zakelly, Thanks for your comments, please find my answers inline. BR, G On Wed, Dec 18, 2024 at 2:42 PM Zakelly Lan wrote: > Hi Gabor, > > Thanks for the FLIP! It's very convenient for users to use SQL to read > checkpoint data, so overall I'd +1 for this FLIP. Here are my > thoughts/quest

Re: [DISCUSS] FLIP-496: SQL connector for keyed state data

Hi Gabor, Thanks for the FLIP! It's very convenient for users to use SQL to read checkpoint data, so overall I'd +1 for this FLIP. Here are my thoughts/questions: 1. I see this connector is designed to read the checkpoint/savepoint data, so I'd suggest naming this connector as 'checkpoint'