Re: [EXTERNAL] Re: Memory growth from TimeWindows

2020-05-27 Thread Slotterback, Chris
start. Chris From: Mitch Lloyd Date: Wednesday, May 27, 2020 at 12:00 PM To: Aljoscha Krettek Cc: "user@flink.apache.org" Subject: Re: [EXTERNAL] Re: Memory growth from TimeWindows Chris, What version of Flink are you using? I also have an issue with slow but continual memory g

Re: [EXTERNAL] Re: Memory growth from TimeWindows

2020-05-27 Thread Mitch Lloyd
Chris, What version of Flink are you using? I also have an issue with slow but continual memory growth in a windowing function but it seems like the taskmanager.sh script I'm using already has the -XX+UseG1GC flag set: https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/t

Re: [EXTERNAL] Re: Memory growth from TimeWindows

2020-05-25 Thread Aljoscha Krettek
Just to double check: the issue was resolved by using a different GC? Because the default GC was too "lazy". ;-) Best, Aljoscha On 21.05.20 18:09, Slotterback, Chris wrote: For those who are interested or googling the mail archives in 8 months, the issue was garbage collection related. The d

Re: [EXTERNAL] Re: Memory growth from TimeWindows

2020-05-21 Thread Slotterback, Chris
For those who are interested or googling the mail archives in 8 months, the issue was garbage collection related. The default 1.8 jvm garbage collector (parallel gc) was being lazy in its marking and collection phases and letting the heap build to a level that was causing memory exceptions and

Re: [EXTERNAL] Re: Memory growth from TimeWindows

2020-05-20 Thread Slotterback, Chris
What I've noticed is that heap memory ends up growing linearly with time indefinitely (past 24 hours) until it hits the roof of the allocated heap for the task manager, which leads me to believe I am leaking somewhere. All of my windows have an allowed lateness of 5 minutes, and my watermarks ar

Re: Memory growth from TimeWindows

2020-05-18 Thread Aljoscha Krettek
On 15.05.20 15:17, Slotterback, Chris wrote: My understanding is that while all these windows build their memory state, I can expect heap memory to grow for the 24 hour length of the SlidingEventTimeWindow, and then start to flatten as the t-24hr window frames expire and release back to the JV

Memory growth from TimeWindows

2020-05-15 Thread Slotterback, Chris
Hey Flink users, I wanted to see if I could get some insight on what the heap memory profile of my stream app should look like vs my expectation. My layout consists of a sequence of FlatMaps + Maps, feeding a pair of 5 minute TumblingEventTimeWindows, intervalJoined, into a 24 hour (per 5 minut