Hi all! Thanks a lot, Xintong, for this thorough analysis. Based on your analysis, here are some thoughts:
+1 to change default JVM metaspace size from 128MB to 64MB +1 to change default JVM overhead min size from 128MB to 196MB Concerning the managed memory fraction, I am not sure I would change it, for the following reasons: - We should assume RocksDB will be limited to managed memory by default. This will either be active by default or we would encourage everyone to use this by default, because otherwise it is super hard to reason about the RocksDB footprint. - For standalone, a managed memory fraction of 0.3 is less than half of the managed memory from 1.9. - I am not sure if the managed memory fraction is a value that all users adjust immediately when scaling up the memory during their first try-out phase. I would assume that most users initially only adjust "memory.flink.size" or "memory.process.size". A value of 0.3 will lead to having too large heaps and very little RocksDB / batch memory even when scaling up during the initial exploration. - I agree, though, that 0.5 looks too aggressive, from your benchmarks. So maybe keeping it at 0.4 could work? And one question: Why do we set the Framework Heap by default? Is that so we reduce the managed memory further is less than framework heap would be left from the JVM heap? Best, Stephan On Thu, Jan 9, 2020 at 10:54 AM Xintong Song <tonysong...@gmail.com> wrote: > Hi all, > > As described in FLINK-15145 [1], we decided to tune the default > configuration values of FLIP-49 with more jobs and cases. > > After spending time analyzing and tuning the configurations, I've come > with several findings. To be brief, I would suggest the following changes, > and for more details please take a look at my tuning report [2]. > > - Change default managed memory fraction from 0.4 to 0.3. > - Change default JVM metaspace size from 128MB to 64MB. > - Change default JVM overhead min size from 128MB to 196MB. > > Looking forward to your feedback. > > Thank you~ > > Xintong Song > > > [1] https://issues.apache.org/jira/browse/FLINK-15145 > > [2] > https://docs.google.com/document/d/1-LravhQYUIkXb7rh0XnBB78vSvhp3ecLSAgsiabfVkk/edit?usp=sharing > >
