Hi all, While working on changing process memory to Flink memory in default configuration, Xintong encountered a problem. When -tm option is used to rewrite container memory, basically process memory, it can collide with the default Flink memory. For legacy users it should not be a problem as we adjusted the legacy heap size option to be interpreted differently for standalone and container modes.
One solution could be to say in -tm docs that we rewrite both options under the hood: process and Flink memory, basically unset Flink memory from yaml config. The downside is that this adds more magic. Alternatively, we can keep process memory in default config and, as mentioned before, increase it to maintain the user experience by matching the previous default setting for heap (now Flink in standalone) size. The Flink memory can be mentioned in process memory comment as a simpler alternative which does not require accounting for JVM overhead. The downside is again more confusion while trying out Flink and tuning memory at the same time. On the other hand, if memory already needs to be tuned it should quite quickly lead to the necessity of understanding the memory model in Flink. Best, Andrey On Thu, Jan 9, 2020 at 12:27 PM Stephan Ewen <se...@apache.org> wrote: > Great! Thanks, guys, for the continued effort on this topic! > > On Thu, Jan 9, 2020 at 5:27 AM Xintong Song <tonysong...@gmail.com> wrote: > > > Thanks all for the discussion. I believe we have get consensus on all the > > open questions discussed in this thread. > > > > Since Andrey already create a jira ticket for renaming shuffle memory > > config keys with "taskmanager.memory.network.*", I'll create ticket for > the > > other topic that puts flink.size in flink-conf.yaml. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Wed, Jan 8, 2020 at 9:39 PM Andrey Zagrebin <azagre...@apache.org> > > wrote: > > > > > It also looks to me that we should only swap network and memory in the > > > option names: 'taskmanager.memory.network.*'. > > > There is no strong consensus towards using new 'shuffle' naming so we > can > > > just rename it to 'taskmanager.memory.network.*' as 'shuffle' naming > has > > > never been released. > > > When we have other shuffle services and start advertising more this > > concept > > > in Flink, we could revisit again the whole naming for this concept. > > > https://jira.apache.org/jira/browse/FLINK-15517 > > > > > >
