Hello everyone, I have a question about the spilling behavior of a Flink batch job.
The relevant part is a standard map-reduce, aggregating 4 billion Tuple3<Integer, Integer, Integer> together via a groupBy(0,1).sum(2). And there really doesn't happen much else in the job. The problem is that I don't understand why this job spills to disk. In this example the spilling is not really an issue, but we run the same job with much larger datasets, where we simply run out of disk space. So we're trying to understand better what it spills and what we can do about it. In this example, I am running on AWS EMR (Flink 1.3.1) with a machine with 240GB memory. I tweaked the following parameters: yarn.heap-cutoff-ratio: 0.1 taskmanager.memory.fraction: 0.9 taskmanager.network.numberOfBuffers: 32768 This leads to 170GB Flink Managed Memory which in my opinion should suffice for the amount of data (the amount of data going from the combine to the reduce is roughly 80GB). However, it is spilling over 70GB on disk. Do you have a hint for me why this could be the case and can explain what exactly is written into the state on such a group-reduce? Thank you so much for your input, best regards Konstantin -- Konstantin Gregor * [email protected] TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Müller Amtsgericht München, HRB 135082
