Re: End-to-end lag spikes when closing a large number of panes

2024-03-27 Thread Robert Metzger
Hey Caio, Your analysis of the problem sounds right to me, I don't have a good solution for you :( I’ve validated that CPU profiles show clearAllState using a significant > amount of CPU. Did you use something like async-profiler here? Do you have more info on the breakdown into what used the C

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user
Hey Asimansu, The inputDataStream is a KeyedStream, I forgot to mention that. Caio On Wed, Mar 20, 2024 at 6:56 PM Asimansu Bera wrote: > Hello Caio, > > Based on the pseudocode, there is no keyed function present. Hence, the > window will not be processed parallely . Please check again and re

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Asimansu Bera
Hello Caio, Based on the pseudocode, there is no keyed function present. Hence, the window will not be processed parallely . Please check again and respond back. val windowDataStream = inputDataStream .window(TumblingEventTimeWindows of 1 hour) .trigger(custom trigger) .aggregat