Thanks for the feedback, Stephan and Kurt.
@Stephan
Regarding managed memory fraction,
- It makes sense to keep the default value 0.4, if we assume rocksdb memory
is limited by default.
- AFAIK, currently rocksdb by default does not limit its memory usage. And
I'm positive to change it.
- Personally, I don't like the idea that we the out-of-box experience (for
which we set the default fraction) relies on that users will manually turn
another switch on.
Regarding framework heap memory,
- The major reason we set it by default is, as you mentioned, that to have
a safe net of minimal JVM heap size.
- Also, considering the in progress FLIP-56 (dynamic slot allocation), we
want to reserve some heap memory that will not go into the slot profiles.
That's why we decide the default value according to the heap memory usage
of an empty task executor.
@Kurt
Regarding metaspace,
- This config option ("taskmanager.memory.jvm-metaspace") only takes effect
on TMs. Currently we do not set metaspace size for JM.
- If we have the same metaspace problem on TMs, then yes, changing it from
128M to 64M will make it worse. However, IMO 10T tpc-ds benchmark should
not be considered as out-of-box experience and it makes sense to tune the
configurations for it. I think the smaller metaspace size would be a better
choice for the first trying-out, where a job should not be too complicated,
the TM size could be relative small (e.g. 1g).
Thank you~
Xintong Song
On Tue, Jan 14, 2020 at 9:38 AM Kurt Young <ykt...@gmail.com> wrote:
> HI Xingtong,
>
> IIRC during our tpc-ds 10T benchmark, we have suffered by JM's metaspace
> size and full gc which
> caused by lots of classloadings of source input split. Could you check
> whether changing the default
> value from 128MB to 64MB will make it worse?
>
> Correct me if I misunderstood anything, also cc @Jingsong
>
> Best,
> Kurt
>
>
> On Tue, Jan 14, 2020 at 3:44 AM Stephan Ewen <se...@apache.org> wrote:
>
>> Hi all!
>>
>> Thanks a lot, Xintong, for this thorough analysis. Based on your analysis,
>> here are some thoughts:
>>
>> +1 to change default JVM metaspace size from 128MB to 64MB
>> +1 to change default JVM overhead min size from 128MB to 196MB
>>
>> Concerning the managed memory fraction, I am not sure I would change it,
>> for the following reasons:
>>
>> - We should assume RocksDB will be limited to managed memory by default.
>> This will either be active by default or we would encourage everyone to
>> use
>> this by default, because otherwise it is super hard to reason about the
>> RocksDB footprint.
>> - For standalone, a managed memory fraction of 0.3 is less than half of
>> the managed memory from 1.9.
>> - I am not sure if the managed memory fraction is a value that all users
>> adjust immediately when scaling up the memory during their first try-out
>> phase. I would assume that most users initially only adjust
>> "memory.flink.size" or "memory.process.size". A value of 0.3 will lead to
>> having too large heaps and very little RocksDB / batch memory even when
>> scaling up during the initial exploration.
>> - I agree, though, that 0.5 looks too aggressive, from your benchmarks.
>> So maybe keeping it at 0.4 could work?
>>
>> And one question: Why do we set the Framework Heap by default? Is that so
>> we reduce the managed memory further is less than framework heap would be
>> left from the JVM heap?
>>
>> Best,
>> Stephan
>>
>> On Thu, Jan 9, 2020 at 10:54 AM Xintong Song <tonysong...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > As described in FLINK-15145 [1], we decided to tune the default
>> > configuration values of FLIP-49 with more jobs and cases.
>> >
>> > After spending time analyzing and tuning the configurations, I've come
>> > with several findings. To be brief, I would suggest the following
>> changes,
>> > and for more details please take a look at my tuning report [2].
>> >
>> > - Change default managed memory fraction from 0.4 to 0.3.
>> > - Change default JVM metaspace size from 128MB to 64MB.
>> > - Change default JVM overhead min size from 128MB to 196MB.
>> >
>> > Looking forward to your feedback.
>> >
>> > Thank you~
>> >
>> > Xintong Song
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-15145
>> >
>> > [2]
>> >
>> https://docs.google.com/document/d/1-LravhQYUIkXb7rh0XnBB78vSvhp3ecLSAgsiabfVkk/edit?usp=sharing
>> >
>> >
>>
>
