onnerstag, 21. Oktober 2021 15:32
To: Schwalbe Matthias
Cc: user
Subject: Re: Huge backpressure when using AggregateFunction with Session Window
Thanks for taking the time to answer this.
* You're correct that the SimpleAggregator is not used in the job setup. I
didn't copy th
Thanks for taking the time to answer this.
- You're correct that the SimpleAggregator is not used in the job setup.
I didn't copy the correct piece of code.
- I understand the overhead involved. But I do not agree with the O(n^2)
complexity. Are you implying that Vector append is O(n)
I didn't try to reproduce it locally since this job reads 14K events per
second.
I am using Flink version 1.12.1 and RocksDB state backend. It also happens
with Flink 1.10.
I tried to profile with JVisualVM and I didn't see any bottleneck. All the
user functions almost didn't take any CPU time.
O
Hi Ori,
Just a couple of comments (some code is missing for a concise explanation):
* SimpleAggregator is not used in the job setup below (assuming another job
setup)
* SimpleAggregator is called for each event that goes into a specific
session window, however
* The scala vectors
Hi Ori,
this sounds indeed strange. Can you also reproduce this behavior locally
with a faker source? We should definitely add a profiler and see where
the bottleneck lies.
Which Flink version and state backend are you using?
Regards,
Timo
On 20.10.21 16:17, Ori Popowski wrote:
I have a sim
