Re: Flink SQL State

2023-04-27 Thread Giannis Polyzos
Will definitely do as it's going to be part of a wider Flink course / book (haven't decided yet on the format) Im putting together. but I can share before that If you want On Thu, Apr 27, 2023 at 6:11 PM Yaroslav Tkachenko wrote: > Got it! Any chance you can open-source some of that? I think it

Re: Flink SQL State

2023-04-27 Thread Yaroslav Tkachenko
Got it! Any chance you can open-source some of that? I think it can be extremely useful for the community. Thank you. On Thu, Apr 27, 2023 at 8:08 AM Giannis Polyzos wrote: > Correct, its some custom code i put together to investigate what gets > written in rocksdb > > On Thu, Apr 27, 2023 at 6

Re: Flink SQL State

2023-04-27 Thread Giannis Polyzos
Correct, its some custom code i put together to investigate what gets written in rocksdb On Thu, Apr 27, 2023 at 6:06 PM Yaroslav Tkachenko wrote: > Hi Giannis, > > I'm curious, what tool did you use for this analysis (what the screenshot > shows)? Is it something custom? > > Thank you. > > On W

Re: Flink SQL State

2023-04-27 Thread Yaroslav Tkachenko
Hi Giannis, I'm curious, what tool did you use for this analysis (what the screenshot shows)? Is it something custom? Thank you. On Wed, Apr 26, 2023 at 10:38 PM Giannis Polyzos wrote: > This is really helpful, > > Thanks > > On Thu, Apr 27, 2023 at 5:46 AM Yanfei Lei wrote: > >> Hi Giannis,

Re: Flink SQL State

2023-04-26 Thread Giannis Polyzos
This is really helpful, Thanks On Thu, Apr 27, 2023 at 5:46 AM Yanfei Lei wrote: > Hi Giannis, > > Except “default” Colume Family(CF), all other CFs represent the state > in rocksdb state backend, the name of a CF is the name of a > StateDescriptor. > > - deduplicate-state is a value state, you

Re: Flink SQL State

2023-04-26 Thread Yanfei Lei
Hi Giannis, Except “default” Colume Family(CF), all other CFs represent the state in rocksdb state backend, the name of a CF is the name of a StateDescriptor. - deduplicate-state is a value state, you can find it in DeduplicateFunctionBase.java and MiniBatchDeduplicateFunctionBase.java, they are

Re: Frequently checkpoint failure, could make the flink sql state not clear?

2020-01-16 Thread Congxian Qiu
Hi, AFAIK, whether a timer will fire is irrelevant to checkpoint success or not. Best, Congxian LakeShen 于2020年1月16日周四 下午8:53写道: > Hi community, now I am using Flink sql , and I set the retention time, As > I all know is that Flink will set the timer for per key to clear their > state, if Fli

Frequently checkpoint failure, could make the flink sql state not clear?

2020-01-16 Thread LakeShen
Hi community, now I am using Flink sql , and I set the retention time, As I all know is that Flink will set the timer for per key to clear their state, if Flink task always checkpoint failure, are the key state cleared by timer? Thanks to your replay.