Re: Flink SQL State

2023-04-27 Thread Giannis Polyzos
Will definitely do as it's going to be part of a wider Flink course / book (haven't decided yet on the format) Im putting together. but I can share before that If you want On Thu, Apr 27, 2023 at 6:11 PM Yaroslav Tkachenko wrote: > Got it! Any chance you can open-source some of that? I think it

Re: Flink SQL State

2023-04-27 Thread Yaroslav Tkachenko
Got it! Any chance you can open-source some of that? I think it can be extremely useful for the community. Thank you. On Thu, Apr 27, 2023 at 8:08 AM Giannis Polyzos wrote: > Correct, its some custom code i put together to investigate what gets > written in rocksdb > > On Thu, Apr 27, 2023 at 6

Re: Flink SQL State

2023-04-27 Thread Giannis Polyzos
Correct, its some custom code i put together to investigate what gets written in rocksdb On Thu, Apr 27, 2023 at 6:06 PM Yaroslav Tkachenko wrote: > Hi Giannis, > > I'm curious, what tool did you use for this analysis (what the screenshot > shows)? Is it something custom? > > Thank you. > > On W

Re: Flink SQL State

2023-04-27 Thread Yaroslav Tkachenko
Hi Giannis, I'm curious, what tool did you use for this analysis (what the screenshot shows)? Is it something custom? Thank you. On Wed, Apr 26, 2023 at 10:38 PM Giannis Polyzos wrote: > This is really helpful, > > Thanks > > On Thu, Apr 27, 2023 at 5:46 AM Yanfei Lei wrote: > >> Hi Giannis,

Re: Flink SQL State

2023-04-26 Thread Giannis Polyzos
This is really helpful, Thanks On Thu, Apr 27, 2023 at 5:46 AM Yanfei Lei wrote: > Hi Giannis, > > Except “default” Colume Family(CF), all other CFs represent the state > in rocksdb state backend, the name of a CF is the name of a > StateDescriptor. > > - deduplicate-state is a value state, you

Re: Flink SQL State

2023-04-26 Thread Yanfei Lei
Hi Giannis, Except “default” Colume Family(CF), all other CFs represent the state in rocksdb state backend, the name of a CF is the name of a StateDescriptor. - deduplicate-state is a value state, you can find it in DeduplicateFunctionBase.java and MiniBatchDeduplicateFunctionBase.java, they are