On 15.05.20 15:17, Slotterback, Chris wrote:
My understanding is that while all these windows build their memory state, I can expect heap memory to grow for the 24 hour length of the SlidingEventTimeWindow, and then start to flatten as the t-24hr window frames expire and release back to the JVM. What is actually happening is when a constant data source feeds the stream, the heap memory profile grows linearly past the 24 hour mark. Could this be a result of a misunderstanding of how the window’s memory states are kept, or is my assumption correct, and it is more likely I have a leak somewhere?
Will memory keep growing indefinitely? That would indicate a bug? What sort of lateness/watermark settings do you have? What window function do you use? ProcessWindowFunction, or sth that aggregates?
Side note: with sliding windows of 24h/5min you will have a "write amplification" of 24*60/5=288, each record will be in 288 windows, which will each be kept in separate state?
Best, Aljoscha
