Re: End-to-end lag spikes when closing a large number of panes

Hey Caio, Your analysis of the problem sounds right to me, I don't have a good solution for you :( I’ve validated that CPU profiles show clearAllState using a significant > amount of CPU. Did you use something like async-profiler here? Do you have more info on the breakdown into what used the C

Re: End-to-end lag spikes when closing a large number of panes

Hey Asimansu, The inputDataStream is a KeyedStream, I forgot to mention that. Caio On Wed, Mar 20, 2024 at 6:56 PM Asimansu Bera wrote: > Hello Caio, > > Based on the pseudocode, there is no keyed function present. Hence, the > window will not be processed parallely . Please check again and re

Re: End-to-end lag spikes when closing a large number of panes

Hello Caio, Based on the pseudocode, there is no keyed function present. Hence, the window will not be processed parallely . Please check again and respond back. val windowDataStream = inputDataStream .window(TumblingEventTimeWindows of 1 hour) .trigger(custom trigger) .aggregat

End-to-end lag spikes when closing a large number of panes

We run a large-scale Flink 1.16 cluster that uses windowed aggregations and we’re seeing lag spikes on window closure. I’m curious if others have encountered similar issues before and if anyone has suggestions for how to tackle this problem (other than simply increasing parallelism). ContextLag def