I also agree that all the configuration should be calculated out of TaskManager.
So a full configuration should be generated before TaskManager started. Override the calculated configurations through -D now seems better. Best, Yang Xintong Song <tonysong...@gmail.com> 于2019年9月2日周一 上午11:39写道： > I just updated the FLIP wiki page [1], with the following changes: > > - Network memory uses JVM direct memory, and is accounted when setting > JVM max direct memory size parameter. > - Use dynamic configurations (`-Dkey=value`) to pass calculated memory > configs into TaskExecutors, instead of ENV variables. > - Remove 'supporting memory reservation' from the scope of this FLIP. > > @till @stephan, please take another look see if there are any other > concerns. > > Thank you~ > > Xintong Song > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > Sorry for the late response. > > > > - Regarding the `TaskExecutorSpecifics` naming, let's discuss the detail > > in PR. > > - Regarding passing parameters into the `TaskExecutor`, +1 for using > > dynamic configuration at the moment, given that there are more questions > to > > be discussed to have a general framework for overwriting configurations > > with ENV variables. > > - Regarding memory reservation, I double checked with Yu and he will take > > care of it. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > >> What I forgot to add is that we could tackle specifying the > configuration > >> fully in an incremental way and that the full specification should be > the > >> desired end state. > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >> > >> > I think our goal should be that the configuration is fully specified > >> when > >> > the process is started. By considering the internal calculation step > to > >> be > >> > rather validate existing values and calculate missing ones, these two > >> > proposal shouldn't even conflict (given determinism). > >> > > >> > Since we don't want to change an existing flink-conf.yaml, specifying > >> the > >> > full configuration would require to pass in the options differently. > >> > > >> > One way could be the ENV variables approach. The reason why I'm trying > >> to > >> > exclude this feature from the FLIP is that I believe it needs a bit > more > >> > discussion. Just some questions which come to my mind: What would be > the > >> > exact format (FLINK_KEY_NAME)? Would we support a dot separator which > is > >> > supported by some systems (FLINK.KEY.NAME)? If we accept the dot > >> > separator what would be the order of precedence if there are two ENV > >> > variables defined (FLINK_KEY_NAME and FLINK.KEY.NAME)? What is the > >> > precedence of env variable vs. dynamic configuration value specified > >> via -D? > >> > > >> > Another approach could be to pass in the dynamic configuration values > >> via > >> > `-Dkey=value` to the Flink process. For that we don't have to change > >> > anything because the functionality already exists. > >> > > >> > Cheers, > >> > Till > >> > > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen <se...@apache.org> > wrote: > >> > > >> >> I see. Under the assumption of strict determinism that should work. > >> >> > >> >> The original proposal had this point "don't compute inside the TM, > >> compute > >> >> outside and supply a full config", because that sounded more > intuitive. > >> >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann <trohrm...@apache.org > > > >> >> wrote: > >> >> > >> >> > My understanding was that before starting the Flink process we > call a > >> >> > utility which calculates these values. I assume that this utility > >> will > >> >> do > >> >> > the calculation based on a set of configured values (process > memory, > >> >> flink > >> >> > memory, network memory etc.). Assuming that these values don't > differ > >> >> from > >> >> > the values with which the JVM is started, it should be possible to > >> >> > recompute them in the Flink process in order to set the values. > >> >> > > >> >> > > >> >> > > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen <se...@apache.org> > >> wrote: > >> >> > > >> >> > > When computing the values in the JVM process after it started, > how > >> >> would > >> >> > > you deal with values like Max Direct Memory, Metaspace size. > native > >> >> > memory > >> >> > > reservation (reduce heap size), etc? All the values that are > >> >> parameters > >> >> > to > >> >> > > the JVM process and that need to be supplied at process startup? > >> >> > > > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann < > >> trohrm...@apache.org> > >> >> > > wrote: > >> >> > > > >> >> > > > Thanks for the clarification. I have some more comments: > >> >> > > > > >> >> > > > - I would actually split the logic to compute the process > memory > >> >> > > > requirements and storing the values into two things. E.g. one > >> could > >> >> > name > >> >> > > > the former TaskExecutorProcessUtility and the latter > >> >> > > > TaskExecutorProcessMemory. But we can discuss this on the PR > >> since > >> >> it's > >> >> > > > just a naming detail. > >> >> > > > > >> >> > > > - Generally, I'm not opposed to making configuration values > >> >> overridable > >> >> > > by > >> >> > > > ENV variables. I think this is a very good idea and makes the > >> >> > > > configurability of Flink processes easier. However, I think > that > >> >> adding > >> >> > > > this functionality should not be part of this FLIP because it > >> would > >> >> > > simply > >> >> > > > widen the scope unnecessarily. > >> >> > > > > >> >> > > > The reasons why I believe it is unnecessary are the following: > >> For > >> >> Yarn > >> >> > > we > >> >> > > > already create write a flink-conf.yaml which could be populated > >> with > >> >> > the > >> >> > > > memory settings. For the other processes it should not make a > >> >> > difference > >> >> > > > whether the loaded Configuration is populated with the memory > >> >> settings > >> >> > > from > >> >> > > > ENV variables or by using TaskExecutorProcessUtility to compute > >> the > >> >> > > missing > >> >> > > > values from the loaded configuration. If the latter would not > be > >> >> > possible > >> >> > > > (wrong or missing configuration values), then we should not > have > >> >> been > >> >> > > able > >> >> > > > to actually start the process in the first place. > >> >> > > > > >> >> > > > - Concerning the memory reservation: I agree with you that we > >> need > >> >> the > >> >> > > > memory reservation functionality to make streaming jobs work > with > >> >> > > "managed" > >> >> > > > memory. However, w/o this functionality the whole Flip would > >> already > >> >> > > bring > >> >> > > > a good amount of improvements to our users when running batch > >> jobs. > >> >> > > > Moreover, by keeping the scope smaller we can complete the FLIP > >> >> faster. > >> >> > > > Hence, I would propose to address the memory reservation > >> >> functionality > >> >> > > as a > >> >> > > > follow up FLIP (which Yu is working on if I'm not mistaken). > >> >> > > > > >> >> > > > Cheers, > >> >> > > > Till > >> >> > > > > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang < > >> danrtsey...@gmail.com> > >> >> > > wrote: > >> >> > > > > >> >> > > > > Just add my 2 cents. > >> >> > > > > > >> >> > > > > Using environment variables to override the configuration for > >> >> > different > >> >> > > > > taskmanagers is better. > >> >> > > > > We do not need to generate dedicated flink-conf.yaml for all > >> >> > > > taskmanagers. > >> >> > > > > A common flink-conf.yam and different environment variables > are > >> >> > enough. > >> >> > > > > By reducing the distributed cached files, it could make > >> launching > >> >> a > >> >> > > > > taskmanager faster. > >> >> > > > > > >> >> > > > > Stephan gives a good suggestion that we could move the logic > >> into > >> >> > > > > "GlobalConfiguration.loadConfig()" method. > >> >> > > > > Maybe the client could also benefit from this. Different > users > >> do > >> >> not > >> >> > > > have > >> >> > > > > to export FLINK_CONF_DIR to update few config options. > >> >> > > > > > >> >> > > > > > >> >> > > > > Best, > >> >> > > > > Yang > >> >> > > > > > >> >> > > > > Stephan Ewen <se...@apache.org> 于2019年8月28日周三 上午1:21写道： > >> >> > > > > > >> >> > > > > > One note on the Environment Variables and Configuration > >> >> discussion. > >> >> > > > > > > >> >> > > > > > My understanding is that passed ENV variables are added to > >> the > >> >> > > > > > configuration in the "GlobalConfiguration.loadConfig()" > >> method > >> >> (or > >> >> > > > > > similar). > >> >> > > > > > For all the code inside Flink, it looks like the data was > in > >> the > >> >> > > config > >> >> > > > > to > >> >> > > > > > start with, just that the scripts that compute the > variables > >> can > >> >> > pass > >> >> > > > the > >> >> > > > > > values to the process without actually needing to write a > >> file. > >> >> > > > > > > >> >> > > > > > For example the "GlobalConfiguration.loadConfig()" method > >> would > >> >> > take > >> >> > > > any > >> >> > > > > > ENV variable prefixed with "flink" and add it as a config > >> key. > >> >> > > > > > "flink_taskmanager_memory_size=2g" would become > >> >> > > > "taskmanager.memory.size: > >> >> > > > > > 2g". > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM Xintong Song < > >> >> > tonysong...@gmail.com> > >> >> > > > > > wrote: > >> >> > > > > > > >> >> > > > > > > Thanks for the comments, Till. > >> >> > > > > > > > >> >> > > > > > > I've also seen your comments on the wiki page, but let's > >> keep > >> >> the > >> >> > > > > > > discussion here. > >> >> > > > > > > > >> >> > > > > > > - Regarding 'TaskExecutorSpecifics', how do you think > about > >> >> > naming > >> >> > > it > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > >> >> > > > > > > - Regarding passing memory configurations into task > >> executors, > >> >> > I'm > >> >> > > in > >> >> > > > > > favor > >> >> > > > > > > of do it via environment variables rather than > >> configurations, > >> >> > with > >> >> > > > the > >> >> > > > > > > following two reasons. > >> >> > > > > > > - It is easier to keep the memory options once > calculate > >> >> not to > >> >> > > be > >> >> > > > > > > changed with environment variables rather than > >> configurations. > >> >> > > > > > > - I'm not sure whether we should write the > configuration > >> in > >> >> > > startup > >> >> > > > > > > scripts. Writing changes into the configuration files > when > >> >> > running > >> >> > > > the > >> >> > > > > > > startup scripts does not sounds right to me. Or we could > >> make > >> >> a > >> >> > > copy > >> >> > > > of > >> >> > > > > > > configuration files per flink cluster, and make the task > >> >> executor > >> >> > > to > >> >> > > > > load > >> >> > > > > > > from the copy, and clean up the copy after the cluster is > >> >> > shutdown, > >> >> > > > > which > >> >> > > > > > > is complicated. (I think this is also what Stephan means > in > >> >> his > >> >> > > > comment > >> >> > > > > > on > >> >> > > > > > > the wiki page?) > >> >> > > > > > > - Regarding reserving memory, I think this change should > be > >> >> > > included > >> >> > > > in > >> >> > > > > > > this FLIP. I think a big part of motivations of this FLIP > >> is > >> >> to > >> >> > > unify > >> >> > > > > > > memory configuration for streaming / batch and make it > easy > >> >> for > >> >> > > > > > configuring > >> >> > > > > > > rocksdb memory. If we don't support memory reservation, > >> then > >> >> > > > streaming > >> >> > > > > > jobs > >> >> > > > > > > cannot use managed memory (neither on-heap or off-heap), > >> which > >> >> > > makes > >> >> > > > > this > >> >> > > > > > > FLIP incomplete. > >> >> > > > > > > - Regarding network memory, I think you are right. I > think > >> we > >> >> > > > probably > >> >> > > > > > > don't need to change network stack from using direct > >> memory to > >> >> > > using > >> >> > > > > > unsafe > >> >> > > > > > > native memory. Network memory size is deterministic, > >> cannot be > >> >> > > > reserved > >> >> > > > > > as > >> >> > > > > > > managed memory does, and cannot be overused. I think it > >> also > >> >> > works > >> >> > > if > >> >> > > > > we > >> >> > > > > > > simply keep using direct memory for network and include > it > >> in > >> >> jvm > >> >> > > max > >> >> > > > > > > direct memory size. > >> >> > > > > > > > >> >> > > > > > > Thank you~ > >> >> > > > > > > > >> >> > > > > > > Xintong Song > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann < > >> >> > > trohrm...@apache.org> > >> >> > > > > > > wrote: > >> >> > > > > > > > >> >> > > > > > > > Hi Xintong, > >> >> > > > > > > > > >> >> > > > > > > > thanks for addressing the comments and adding a more > >> >> detailed > >> >> > > > > > > > implementation plan. I have a couple of comments > >> concerning > >> >> the > >> >> > > > > > > > implementation plan: > >> >> > > > > > > > > >> >> > > > > > > > - The name `TaskExecutorSpecifics` is not really > >> >> descriptive. > >> >> > > > > Choosing > >> >> > > > > > a > >> >> > > > > > > > different name could help here. > >> >> > > > > > > > - I'm not sure whether I would pass the memory > >> >> configuration to > >> >> > > the > >> >> > > > > > > > TaskExecutor via environment variables. I think it > would > >> be > >> >> > > better > >> >> > > > to > >> >> > > > > > > write > >> >> > > > > > > > it into the configuration one uses to start the TM > >> process. > >> >> > > > > > > > - If possible, I would exclude the memory reservation > >> from > >> >> this > >> >> > > > FLIP > >> >> > > > > > and > >> >> > > > > > > > add this as part of a dedicated FLIP. > >> >> > > > > > > > - If possible, then I would exclude changes to the > >> network > >> >> > stack > >> >> > > > from > >> >> > > > > > > this > >> >> > > > > > > > FLIP. Maybe we can simply say that the direct memory > >> needed > >> >> by > >> >> > > the > >> >> > > > > > > network > >> >> > > > > > > > stack is the framework direct memory requirement. > >> Changing > >> >> how > >> >> > > the > >> >> > > > > > memory > >> >> > > > > > > > is allocated can happen in a second step. This would > keep > >> >> the > >> >> > > scope > >> >> > > > > of > >> >> > > > > > > this > >> >> > > > > > > > FLIP smaller. > >> >> > > > > > > > > >> >> > > > > > > > Cheers, > >> >> > > > > > > > Till > >> >> > > > > > > > > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM Xintong Song < > >> >> > > > tonysong...@gmail.com> > >> >> > > > > > > > wrote: > >> >> > > > > > > > > >> >> > > > > > > > > Hi everyone, > >> >> > > > > > > > > > >> >> > > > > > > > > I just updated the FLIP document on wiki [1], with > the > >> >> > > following > >> >> > > > > > > changes. > >> >> > > > > > > > > > >> >> > > > > > > > > - Removed open question regarding MemorySegment > >> >> > allocation. > >> >> > > As > >> >> > > > > > > > > discussed, we exclude this topic from the scope of > >> this > >> >> > > FLIP. > >> >> > > > > > > > > - Updated content about JVM direct memory > parameter > >> >> > > according > >> >> > > > to > >> >> > > > > > > > recent > >> >> > > > > > > > > discussions, and moved the other options to > >> "Rejected > >> >> > > > > > Alternatives" > >> >> > > > > > > > for > >> >> > > > > > > > > the > >> >> > > > > > > > > moment. > >> >> > > > > > > > > - Added implementation steps. > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > Thank you~ > >> >> > > > > > > > > > >> >> > > > > > > > > Xintong Song > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > [1] > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >> >> > > > > > > > > > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM Stephan Ewen < > >> >> > se...@apache.org > >> >> > > > > >> >> > > > > > wrote: > >> >> > > > > > > > > > >> >> > > > > > > > > > @Xintong: Concerning "wait for memory users before > >> task > >> >> > > dispose > >> >> > > > > and > >> >> > > > > > > > > memory > >> >> > > > > > > > > > release": I agree, that's how it should be. Let's > >> try it > >> >> > out. > >> >> > > > > > > > > > > >> >> > > > > > > > > > @Xintong @Jingsong: Concerning " JVM does not wait > >> for > >> >> GC > >> >> > > when > >> >> > > > > > > > allocating > >> >> > > > > > > > > > direct memory buffer": There seems to be pretty > >> >> elaborate > >> >> > > logic > >> >> > > > > to > >> >> > > > > > > free > >> >> > > > > > > > > > buffers when allocating new ones. See > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > >> >> > > > > > > > > > > >> >> > > > > > > > > > @Till: Maybe. If we assume that the JVM default > works > >> >> (like > >> >> > > > going > >> >> > > > > > > with > >> >> > > > > > > > > > option 2 and not setting "-XX:MaxDirectMemorySize" > at > >> >> all), > >> >> > > > then > >> >> > > > > I > >> >> > > > > > > > think > >> >> > > > > > > > > it > >> >> > > > > > > > > > should be okay to set "-XX:MaxDirectMemorySize" to > >> >> > > > > > > > > > "off_heap_managed_memory + direct_memory" even if > we > >> use > >> >> > > > RocksDB. > >> >> > > > > > > That > >> >> > > > > > > > > is a > >> >> > > > > > > > > > big if, though, I honestly have no idea :D Would be > >> >> good to > >> >> > > > > > > understand > >> >> > > > > > > > > > this, though, because this would affect option (2) > >> and > >> >> > option > >> >> > > > > > (1.2). > >> >> > > > > > > > > > > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 PM Xintong Song < > >> >> > > > > > tonysong...@gmail.com> > >> >> > > > > > > > > > wrote: > >> >> > > > > > > > > > > >> >> > > > > > > > > > > Thanks for the inputs, Jingsong. > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > Let me try to summarize your points. Please > correct > >> >> me if > >> >> > > I'm > >> >> > > > > > > wrong. > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > - Memory consumers should always avoid > returning > >> >> > memory > >> >> > > > > > segments > >> >> > > > > > > > to > >> >> > > > > > > > > > > memory manager while there are still > un-cleaned > >> >> > > > structures / > >> >> > > > > > > > threads > >> >> > > > > > > > > > > that > >> >> > > > > > > > > > > may use the memory. Otherwise, it would cause > >> >> serious > >> >> > > > > problems > >> >> > > > > > > by > >> >> > > > > > > > > > having > >> >> > > > > > > > > > > multiple consumers trying to use the same > memory > >> >> > > segment. > >> >> > > > > > > > > > > - JVM does not wait for GC when allocating > >> direct > >> >> > memory > >> >> > > > > > buffer. > >> >> > > > > > > > > > > Therefore even we set proper max direct memory > >> size > >> >> > > limit, > >> >> > > > > we > >> >> > > > > > > may > >> >> > > > > > > > > > still > >> >> > > > > > > > > > > encounter direct memory oom if the GC cleaning > >> >> memory > >> >> > > > slower > >> >> > > > > > > than > >> >> > > > > > > > > the > >> >> > > > > > > > > > > direct memory allocation. > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > Am I understanding this correctly? > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 PM JingsongLee < > >> >> > > > > > > lzljs3620...@aliyun.com > >> >> > > > > > > > > > > .invalid> > >> >> > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > Hi stephan: > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > About option 2: > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > if additional threads not cleanly shut down > >> before > >> >> we > >> >> > can > >> >> > > > > exit > >> >> > > > > > > the > >> >> > > > > > > > > > task: > >> >> > > > > > > > > > > > In the current case of memory reuse, it has > >> freed up > >> >> > the > >> >> > > > > memory > >> >> > > > > > > it > >> >> > > > > > > > > > > > uses. If this memory is used by other tasks > and > >> >> > > > asynchronous > >> >> > > > > > > > threads > >> >> > > > > > > > > > > > of exited task may still be writing, there > will > >> be > >> >> > > > > concurrent > >> >> > > > > > > > > security > >> >> > > > > > > > > > > > problems, and even lead to errors in user > >> computing > >> >> > > > results. > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > So I think this is a serious and intolerable > >> bug, No > >> >> > > matter > >> >> > > > > > what > >> >> > > > > > > > the > >> >> > > > > > > > > > > > option is, it should be avoided. > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > About direct memory cleaned by GC: > >> >> > > > > > > > > > > > I don't think it is a good idea, I've > >> encountered so > >> >> > many > >> >> > > > > > > > situations > >> >> > > > > > > > > > > > that it's too late for GC to cause > DirectMemory > >> >> OOM. > >> >> > > > Release > >> >> > > > > > and > >> >> > > > > > > > > > > > allocate DirectMemory depend on the type of > user > >> >> job, > >> >> > > > which > >> >> > > > > is > >> >> > > > > > > > > > > > often beyond our control. > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > Best, > >> >> > > > > > > > > > > > Jingsong Lee > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > >> >> > ------------------------------------------------------------------ > >> >> > > > > > > > > > > > From:Stephan Ewen <se...@apache.org> > >> >> > > > > > > > > > > > Send Time:2019年8月19日(星期一) 15:56 > >> >> > > > > > > > > > > > To:dev <dev@flink.apache.org> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] FLIP-49: Unified Memory > >> >> > > Configuration > >> >> > > > > for > >> >> > > > > > > > > > > > TaskExecutors > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > My main concern with option 2 (manually release > >> >> memory) > >> >> > > is > >> >> > > > > that > >> >> > > > > > > > > > segfaults > >> >> > > > > > > > > > > > in the JVM send off all sorts of alarms on user > >> >> ends. > >> >> > So > >> >> > > we > >> >> > > > > > need > >> >> > > > > > > to > >> >> > > > > > > > > > > > guarantee that this never happens. > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > The trickyness is in tasks that uses data > >> >> structures / > >> >> > > > > > algorithms > >> >> > > > > > > > > with > >> >> > > > > > > > > > > > additional threads, like hash table spill/read > >> and > >> >> > > sorting > >> >> > > > > > > threads. > >> >> > > > > > > > > We > >> >> > > > > > > > > > > need > >> >> > > > > > > > > > > > to ensure that these cleanly shut down before > we > >> can > >> >> > exit > >> >> > > > the > >> >> > > > > > > task. > >> >> > > > > > > > > > > > I am not sure that we have that guaranteed > >> already, > >> >> > > that's > >> >> > > > > why > >> >> > > > > > > > option > >> >> > > > > > > > > > 1.1 > >> >> > > > > > > > > > > > seemed simpler to me. > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at 3:42 PM Xintong Song < > >> >> > > > > > > > tonysong...@gmail.com> > >> >> > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Thanks for the comments, Stephan. Summarized > in > >> >> this > >> >> > > way > >> >> > > > > > really > >> >> > > > > > > > > makes > >> >> > > > > > > > > > > > > things easier to understand. > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > I'm in favor of option 2, at least for the > >> >> moment. I > >> >> > > > think > >> >> > > > > it > >> >> > > > > > > is > >> >> > > > > > > > > not > >> >> > > > > > > > > > > that > >> >> > > > > > > > > > > > > difficult to keep it segfault safe for memory > >> >> > manager, > >> >> > > as > >> >> > > > > > long > >> >> > > > > > > as > >> >> > > > > > > > > we > >> >> > > > > > > > > > > > always > >> >> > > > > > > > > > > > > de-allocate the memory segment when it is > >> released > >> >> > from > >> >> > > > the > >> >> > > > > > > > memory > >> >> > > > > > > > > > > > > consumers. Only if the memory consumer > continue > >> >> using > >> >> > > the > >> >> > > > > > > buffer > >> >> > > > > > > > of > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > segment after releasing it, in which case we > do > >> >> want > >> >> > > the > >> >> > > > > job > >> >> > > > > > to > >> >> > > > > > > > > fail > >> >> > > > > > > > > > so > >> >> > > > > > > > > > > > we > >> >> > > > > > > > > > > > > detect the memory leak early. > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > For option 1.2, I don't think this is a good > >> idea. > >> >> > Not > >> >> > > > only > >> >> > > > > > > > because > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > assumption (regular GC is enough to clean > >> direct > >> >> > > buffers) > >> >> > > > > may > >> >> > > > > > > not > >> >> > > > > > > > > > > always > >> >> > > > > > > > > > > > be > >> >> > > > > > > > > > > > > true, but also it makes harder for finding > >> >> problems > >> >> > in > >> >> > > > > cases > >> >> > > > > > of > >> >> > > > > > > > > > memory > >> >> > > > > > > > > > > > > overuse. E.g., user configured some direct > >> memory > >> >> for > >> >> > > the > >> >> > > > > > user > >> >> > > > > > > > > > > libraries. > >> >> > > > > > > > > > > > > If the library actually use more direct > memory > >> >> then > >> >> > > > > > configured, > >> >> > > > > > > > > which > >> >> > > > > > > > > > > > > cannot be cleaned by GC because they are > still > >> in > >> >> > use, > >> >> > > > may > >> >> > > > > > lead > >> >> > > > > > > > to > >> >> > > > > > > > > > > > overuse > >> >> > > > > > > > > > > > > of the total container memory. In that case, > >> if it > >> >> > > didn't > >> >> > > > > > touch > >> >> > > > > > > > the > >> >> > > > > > > > > > JVM > >> >> > > > > > > > > > > > > default max direct memory limit, we cannot > get > >> a > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > OOM > >> >> > > > > > > > > > and > >> >> > > > > > > > > > > it > >> >> > > > > > > > > > > > > will become super hard to understand which > >> part of > >> >> > the > >> >> > > > > > > > > configuration > >> >> > > > > > > > > > > need > >> >> > > > > > > > > > > > > to be updated. > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > For option 1.1, it has the similar problem as > >> >> 1.2, if > >> >> > > the > >> >> > > > > > > > exceeded > >> >> > > > > > > > > > > direct > >> >> > > > > > > > > > > > > memory does not reach the max direct memory > >> limit > >> >> > > > specified > >> >> > > > > > by > >> >> > > > > > > > the > >> >> > > > > > > > > > > > > dedicated parameter. I think it is slightly > >> better > >> >> > than > >> >> > > > > 1.2, > >> >> > > > > > > only > >> >> > > > > > > > > > > because > >> >> > > > > > > > > > > > > we can tune the parameter. > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen > < > >> >> > > > > > se...@apache.org > >> >> > > > > > > > > >> >> > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > About the "-XX:MaxDirectMemorySize" > >> discussion, > >> >> > maybe > >> >> > > > let > >> >> > > > > > me > >> >> > > > > > > > > > > summarize > >> >> > > > > > > > > > > > > it a > >> >> > > > > > > > > > > > > > bit differently: > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > We have the following two options: > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > (1) We let MemorySegments be de-allocated > by > >> the > >> >> > GC. > >> >> > > > That > >> >> > > > > > > makes > >> >> > > > > > > > > it > >> >> > > > > > > > > > > > > segfault > >> >> > > > > > > > > > > > > > safe. But then we need a way to trigger GC > in > >> >> case > >> >> > > > > > > > de-allocation > >> >> > > > > > > > > > and > >> >> > > > > > > > > > > > > > re-allocation of a bunch of segments > happens > >> >> > quickly, > >> >> > > > > which > >> >> > > > > > > is > >> >> > > > > > > > > > often > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > case during batch scheduling or task > restart. > >> >> > > > > > > > > > > > > > - The "-XX:MaxDirectMemorySize" (option > >> 1.1) > >> >> is > >> >> > one > >> >> > > > way > >> >> > > > > > to > >> >> > > > > > > do > >> >> > > > > > > > > > this > >> >> > > > > > > > > > > > > > - Another way could be to have a > dedicated > >> >> > > > bookkeeping > >> >> > > > > in > >> >> > > > > > > the > >> >> > > > > > > > > > > > > > MemoryManager (option 1.2), so that this > is a > >> >> > number > >> >> > > > > > > > independent > >> >> > > > > > > > > of > >> >> > > > > > > > > > > the > >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" parameter. > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > (2) We manually allocate and de-allocate > the > >> >> memory > >> >> > > for > >> >> > > > > the > >> >> > > > > > > > > > > > > MemorySegments > >> >> > > > > > > > > > > > > > (option 2). That way we need not worry > about > >> >> > > triggering > >> >> > > > > GC > >> >> > > > > > by > >> >> > > > > > > > > some > >> >> > > > > > > > > > > > > > threshold or bookkeeping, but it is harder > to > >> >> > prevent > >> >> > > > > > > > segfaults. > >> >> > > > > > > > > We > >> >> > > > > > > > > > > > need > >> >> > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > be very careful about when we release the > >> memory > >> >> > > > segments > >> >> > > > > > > (only > >> >> > > > > > > > > in > >> >> > > > > > > > > > > the > >> >> > > > > > > > > > > > > > cleanup phase of the main thread). > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > If we go with option 1.1, we probably need > to > >> >> set > >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" to > >> >> > > "off_heap_managed_memory + > >> >> > > > > > > > > > > direct_memory" > >> >> > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > have "direct_memory" as a separate reserved > >> >> memory > >> >> > > > pool. > >> >> > > > > > > > Because > >> >> > > > > > > > > if > >> >> > > > > > > > > > > we > >> >> > > > > > > > > > > > > just > >> >> > > > > > > > > > > > > > set "-XX:MaxDirectMemorySize" to > >> >> > > > > "off_heap_managed_memory + > >> >> > > > > > > > > > > > > jvm_overhead", > >> >> > > > > > > > > > > > > > then there will be times when that entire > >> >> memory is > >> >> > > > > > allocated > >> >> > > > > > > > by > >> >> > > > > > > > > > > direct > >> >> > > > > > > > > > > > > > buffers and we have nothing left for the > JVM > >> >> > > overhead. > >> >> > > > So > >> >> > > > > > we > >> >> > > > > > > > > either > >> >> > > > > > > > > > > > need > >> >> > > > > > > > > > > > > a > >> >> > > > > > > > > > > > > > way to compensate for that (again some > safety > >> >> > margin > >> >> > > > > cutoff > >> >> > > > > > > > > value) > >> >> > > > > > > > > > or > >> >> > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > will exceed container memory. > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > If we go with option 1.2, we need to be > aware > >> >> that > >> >> > it > >> >> > > > > takes > >> >> > > > > > > > > > elaborate > >> >> > > > > > > > > > > > > logic > >> >> > > > > > > > > > > > > > to push recycling of direct buffers without > >> >> always > >> >> > > > > > > triggering a > >> >> > > > > > > > > > full > >> >> > > > > > > > > > > > GC. > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > My first guess is that the options will be > >> >> easiest > >> >> > to > >> >> > > > do > >> >> > > > > in > >> >> > > > > > > the > >> >> > > > > > > > > > > > following > >> >> > > > > > > > > > > > > > order: > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 1.1 with a dedicated > direct_memory > >> >> > > > parameter, > >> >> > > > > as > >> >> > > > > > > > > > discussed > >> >> > > > > > > > > > > > > > above. We would need to find a way to set > the > >> >> > > > > direct_memory > >> >> > > > > > > > > > parameter > >> >> > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > default. We could start with 64 MB and see > >> how > >> >> it > >> >> > > goes > >> >> > > > in > >> >> > > > > > > > > practice. > >> >> > > > > > > > > > > One > >> >> > > > > > > > > > > > > > danger I see is that setting this loo low > can > >> >> > cause a > >> >> > > > > bunch > >> >> > > > > > > of > >> >> > > > > > > > > > > > additional > >> >> > > > > > > > > > > > > > GCs compared to before (we need to watch > this > >> >> > > > carefully). > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 2. It is actually quite simple > to > >> >> > > implement, > >> >> > > > > we > >> >> > > > > > > > could > >> >> > > > > > > > > > try > >> >> > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > segfault safe we are at the moment. > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 1.2: We would not touch the > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > >> >> > > > > > > > > > > > > parameter > >> >> > > > > > > > > > > > > > at all and assume that all the direct > memory > >> >> > > > allocations > >> >> > > > > > that > >> >> > > > > > > > the > >> >> > > > > > > > > > JVM > >> >> > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > Netty do are infrequent enough to be > cleaned > >> up > >> >> > fast > >> >> > > > > enough > >> >> > > > > > > > > through > >> >> > > > > > > > > > > > > regular > >> >> > > > > > > > > > > > > > GC. I am not sure if that is a valid > >> assumption, > >> >> > > > though. > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > Best, > >> >> > > > > > > > > > > > > > Stephan > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong > Song > >> < > >> >> > > > > > > > > > tonysong...@gmail.com> > >> >> > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thanks for sharing your opinion Till. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was > >> >> > wondering > >> >> > > > > > whether > >> >> > > > > > > > we > >> >> > > > > > > > > > can > >> >> > > > > > > > > > > > > avoid > >> >> > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap > >> managed > >> >> > memory > >> >> > > > and > >> >> > > > > > > > network > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > with > >> >> > > > > > > > > > > > > > > alternative 3. But after giving it a > second > >> >> > > thought, > >> >> > > > I > >> >> > > > > > > think > >> >> > > > > > > > > even > >> >> > > > > > > > > > > for > >> >> > > > > > > > > > > > > > > alternative 3 using direct memory for > >> off-heap > >> >> > > > managed > >> >> > > > > > > memory > >> >> > > > > > > > > > could > >> >> > > > > > > > > > > > > cause > >> >> > > > > > > > > > > > > > > problems. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Hi Yang, > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Regarding your concern, I think what > >> proposed > >> >> in > >> >> > > this > >> >> > > > > > FLIP > >> >> > > > > > > it > >> >> > > > > > > > > to > >> >> > > > > > > > > > > have > >> >> > > > > > > > > > > > > > both > >> >> > > > > > > > > > > > > > > off-heap managed memory and network > memory > >> >> > > allocated > >> >> > > > > > > through > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which means they are > >> >> > practically > >> >> > > > > > native > >> >> > > > > > > > > memory > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > limited by JVM max direct memory. The > only > >> >> parts > >> >> > of > >> >> > > > > > memory > >> >> > > > > > > > > > limited > >> >> > > > > > > > > > > by > >> >> > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > max direct memory are task off-heap > memory > >> and > >> >> > JVM > >> >> > > > > > > overhead, > >> >> > > > > > > > > > which > >> >> > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > exactly alternative 2 suggests to set the > >> JVM > >> >> max > >> >> > > > > direct > >> >> > > > > > > > memory > >> >> > > > > > > > > > to. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till > >> Rohrmann > >> >> < > >> >> > > > > > > > > > > trohrm...@apache.org> > >> >> > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I > >> >> > > understand > >> >> > > > > the > >> >> > > > > > > two > >> >> > > > > > > > > > > > > alternatives > >> >> > > > > > > > > > > > > > > > now. > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > I would be in favour of option 2 > because > >> it > >> >> > makes > >> >> > > > > > things > >> >> > > > > > > > > > > explicit. > >> >> > > > > > > > > > > > If > >> >> > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > don't limit the direct memory, I fear > >> that > >> >> we > >> >> > > might > >> >> > > > > end > >> >> > > > > > > up > >> >> > > > > > > > > in a > >> >> > > > > > > > > > > > > similar > >> >> > > > > > > > > > > > > > > > situation as we are currently in: The > >> user > >> >> > might > >> >> > > > see > >> >> > > > > > that > >> >> > > > > > > > her > >> >> > > > > > > > > > > > process > >> >> > > > > > > > > > > > > > > gets > >> >> > > > > > > > > > > > > > > > killed by the OS and does not know why > >> this > >> >> is > >> >> > > the > >> >> > > > > > case. > >> >> > > > > > > > > > > > > Consequently, > >> >> > > > > > > > > > > > > > > she > >> >> > > > > > > > > > > > > > > > tries to decrease the process memory > size > >> >> > > (similar > >> >> > > > to > >> >> > > > > > > > > > increasing > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > cutoff > >> >> > > > > > > > > > > > > > > > ratio) in order to accommodate for the > >> extra > >> >> > > direct > >> >> > > > > > > memory. > >> >> > > > > > > > > > Even > >> >> > > > > > > > > > > > > worse, > >> >> > > > > > > > > > > > > > > she > >> >> > > > > > > > > > > > > > > > tries to decrease memory budgets which > >> are > >> >> not > >> >> > > > fully > >> >> > > > > > used > >> >> > > > > > > > and > >> >> > > > > > > > > > > hence > >> >> > > > > > > > > > > > > > won't > >> >> > > > > > > > > > > > > > > > change the overall memory consumption. > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM > Xintong > >> >> Song < > >> >> > > > > > > > > > > > tonysong...@gmail.com > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let me explain this with a concrete > >> >> example > >> >> > > Till. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let's say we have the following > >> scenario. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Total Process Memory: 1GB > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap > >> Memory + > >> >> JVM > >> >> > > > > > > Overhead): > >> >> > > > > > > > > > 200MB > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM > >> >> Metaspace, > >> >> > > > > > Off-Heap > >> >> > > > > > > > > > Managed > >> >> > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > Network Memory): 800MB > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > For alternative 2, we set > >> >> > > -XX:MaxDirectMemorySize > >> >> > > > > to > >> >> > > > > > > > 200MB. > >> >> > > > > > > > > > > > > > > > > For alternative 3, we set > >> >> > > -XX:MaxDirectMemorySize > >> >> > > > > to > >> >> > > > > > a > >> >> > > > > > > > very > >> >> > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > value, > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of > >> Task > >> >> > > > Off-Heap > >> >> > > > > > > Memory > >> >> > > > > > > > > and > >> >> > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > Overhead > >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, then > alternative 2 > >> >> and > >> >> > > > > > > alternative 3 > >> >> > > > > > > > > > > should > >> >> > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > same utility. Setting larger > >> >> > > > > -XX:MaxDirectMemorySize > >> >> > > > > > > will > >> >> > > > > > > > > not > >> >> > > > > > > > > > > > > reduce > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > sizes of the other memory pools. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of > >> Task > >> >> > > > Off-Heap > >> >> > > > > > > Memory > >> >> > > > > > > > > and > >> >> > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, > then > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > - Alternative 2 suffers from > >> frequent > >> >> OOM. > >> >> > > To > >> >> > > > > > avoid > >> >> > > > > > > > > that, > >> >> > > > > > > > > > > the > >> >> > > > > > > > > > > > > only > >> >> > > > > > > > > > > > > > > > thing > >> >> > > > > > > > > > > > > > > > > user can do is to modify the > >> >> configuration > >> >> > > and > >> >> > > > > > > > increase > >> >> > > > > > > > > > JVM > >> >> > > > > > > > > > > > > Direct > >> >> > > > > > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM > >> Overhead). > >> >> > Let's > >> >> > > > say > >> >> > > > > > > that > >> >> > > > > > > > > user > >> >> > > > > > > > > > > > > > increases > >> >> > > > > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will > >> >> reduce > >> >> > the > >> >> > > > > total > >> >> > > > > > > > size > >> >> > > > > > > > > of > >> >> > > > > > > > > > > > other > >> >> > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > pools to 750MB, given the total > >> process > >> >> > > memory > >> >> > > > > > > remains > >> >> > > > > > > > > > 1GB. > >> >> > > > > > > > > > > > > > > > > - For alternative 3, there is no > >> >> chance of > >> >> > > > > direct > >> >> > > > > > > OOM. > >> >> > > > > > > > > > There > >> >> > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > chances > >> >> > > > > > > > > > > > > > > > > of exceeding the total process > >> memory > >> >> > limit, > >> >> > > > but > >> >> > > > > > > given > >> >> > > > > > > > > > that > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > process > >> >> > > > > > > > > > > > > > > > > may > >> >> > > > > > > > > > > > > > > > > not use up all the reserved native > >> >> memory > >> >> > > > > > (Off-Heap > >> >> > > > > > > > > > Managed > >> >> > > > > > > > > > > > > > Memory, > >> >> > > > > > > > > > > > > > > > > Network > >> >> > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the > >> actual > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > > usage > >> >> > > > > > > > > > is > >> >> > > > > > > > > > > > > > > slightly > >> >> > > > > > > > > > > > > > > > > above > >> >> > > > > > > > > > > > > > > > > yet very close to 200MB, user > >> probably > >> >> do > >> >> > > not > >> >> > > > > need > >> >> > > > > > > to > >> >> > > > > > > > > > change > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > configurations. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Therefore, I think from the user's > >> >> > > perspective, a > >> >> > > > > > > > feasible > >> >> > > > > > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > > > > for alternative 2 may lead to lower > >> >> resource > >> >> > > > > > > utilization > >> >> > > > > > > > > > > compared > >> >> > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > alternative 3. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till > >> >> > Rohrmann > >> >> > > < > >> >> > > > > > > > > > > > > trohrm...@apache.org > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > I guess you have to help me > >> understand > >> >> the > >> >> > > > > > difference > >> >> > > > > > > > > > between > >> >> > > > > > > > > > > > > > > > > alternative 2 > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to memory under > utilization > >> >> > > Xintong. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: set > >> >> XX:MaxDirectMemorySize > >> >> > > to > >> >> > > > > Task > >> >> > > > > > > > > > Off-Heap > >> >> > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > > Overhead. Then there is the risk > that > >> >> this > >> >> > > size > >> >> > > > > is > >> >> > > > > > > too > >> >> > > > > > > > > low > >> >> > > > > > > > > > > > > > resulting > >> >> > > > > > > > > > > > > > > > in a > >> >> > > > > > > > > > > > > > > > > > lot of garbage collection and > >> >> potentially > >> >> > an > >> >> > > > OOM. > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: set > >> >> XX:MaxDirectMemorySize > >> >> > > to > >> >> > > > > > > > something > >> >> > > > > > > > > > > larger > >> >> > > > > > > > > > > > > > than > >> >> > > > > > > > > > > > > > > > > > alternative 2. This would of course > >> >> reduce > >> >> > > the > >> >> > > > > > sizes > >> >> > > > > > > of > >> >> > > > > > > > > the > >> >> > > > > > > > > > > > other > >> >> > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > types. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > How would alternative 2 now result > >> in an > >> >> > > under > >> >> > > > > > > > > utilization > >> >> > > > > > > > > > of > >> >> > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > compared to alternative 3? If > >> >> alternative 3 > >> >> > > > > > strictly > >> >> > > > > > > > > sets a > >> >> > > > > > > > > > > > > higher > >> >> > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > direct memory size and we use only > >> >> little, > >> >> > > > then I > >> >> > > > > > > would > >> >> > > > > > > > > > > expect > >> >> > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > alternative 3 results in memory > under > >> >> > > > > utilization. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM > Yang > >> >> Wang < > >> >> > > > > > > > > > > > danrtsey...@gmail.com > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Native and Direct Memory > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > My point is setting a very large > >> max > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > size > >> >> > > > > > > > > > > when > >> >> > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > do > >> >> > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > differentiate direct and native > >> >> memory. > >> >> > If > >> >> > > > the > >> >> > > > > > > direct > >> >> > > > > > > > > > > > > > > > memory,including > >> >> > > > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > direct memory and framework > direct > >> >> > > > memory,could > >> >> > > > > > be > >> >> > > > > > > > > > > calculated > >> >> > > > > > > > > > > > > > > > > > > correctly,then > >> >> > > > > > > > > > > > > > > > > > > i am in favor of setting direct > >> memory > >> >> > with > >> >> > > > > fixed > >> >> > > > > > > > > value. > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Memory Calculation > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn > and > >> >> k8s,we > >> >> > > > need > >> >> > > > > to > >> >> > > > > > > > check > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > configurations in client to avoid > >> >> > > submitting > >> >> > > > > > > > > successfully > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > failing > >> >> > > > > > > > > > > > > > > > > in > >> >> > > > > > > > > > > > > > > > > > > the flink master. > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Best, > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Yang > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > >> tonysong...@gmail.com > >> >> > > > > >于2019年8月13日 > >> >> > > > > > > > > > 周二22:07写道： > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About MemorySegment, I think > you > >> are > >> >> > > right > >> >> > > > > that > >> >> > > > > > > we > >> >> > > > > > > > > > should > >> >> > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > include > >> >> > > > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > issue in the scope of this > FLIP. > >> >> This > >> >> > > FLIP > >> >> > > > > > should > >> >> > > > > > > > > > > > concentrate > >> >> > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > configure memory pools for > >> >> > TaskExecutors, > >> >> > > > > with > >> >> > > > > > > > > minimum > >> >> > > > > > > > > > > > > > > involvement > >> >> > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > > > > > memory consumers use it. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About direct memory, I think > >> >> > alternative > >> >> > > 3 > >> >> > > > > may > >> >> > > > > > > not > >> >> > > > > > > > > > having > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > same > >> >> > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > reservation issue that > >> alternative 2 > >> >> > > does, > >> >> > > > > but > >> >> > > > > > at > >> >> > > > > > > > the > >> >> > > > > > > > > > > cost > >> >> > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > risk > >> >> > > > > > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > using memory at the container > >> level, > >> >> > > which > >> >> > > > is > >> >> > > > > > not > >> >> > > > > > > > > good. > >> >> > > > > > > > > > > My > >> >> > > > > > > > > > > > > > point > >> >> > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and > >> "JVM > >> >> > > > > Overhead" > >> >> > > > > > > are > >> >> > > > > > > > > not > >> >> > > > > > > > > > > easy > >> >> > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > config. > >> >> > > > > > > > > > > > > > > > > > > For > >> >> > > > > > > > > > > > > > > > > > > > alternative 2, users might > >> configure > >> >> > them > >> >> > > > > > higher > >> >> > > > > > > > than > >> >> > > > > > > > > > > what > >> >> > > > > > > > > > > > > > > actually > >> >> > > > > > > > > > > > > > > > > > > needed, > >> >> > > > > > > > > > > > > > > > > > > > just to avoid getting a direct > >> OOM. > >> >> For > >> >> > > > > > > alternative > >> >> > > > > > > > > 3, > >> >> > > > > > > > > > > > users > >> >> > > > > > > > > > > > > do > >> >> > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > get > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so they may not > >> config > >> >> the > >> >> > > two > >> >> > > > > > > options > >> >> > > > > > > > > > > > > aggressively > >> >> > > > > > > > > > > > > > > > high. > >> >> > > > > > > > > > > > > > > > > > But > >> >> > > > > > > > > > > > > > > > > > > > the consequences are risks of > >> >> overall > >> >> > > > > container > >> >> > > > > > > > > memory > >> >> > > > > > > > > > > > usage > >> >> > > > > > > > > > > > > > > > exceeds > >> >> > > > > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > > > > budget. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM > >> Till > >> >> > > > > Rohrmann < > >> >> > > > > > > > > > > > > > > > trohrm...@apache.org> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for proposing this > FLIP > >> >> > Xintong. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > All in all I think it already > >> >> looks > >> >> > > quite > >> >> > > > > > good. > >> >> > > > > > > > > > > > Concerning > >> >> > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > first > >> >> > > > > > > > > > > > > > > > > > > open > >> >> > > > > > > > > > > > > > > > > > > > > question about allocating > >> memory > >> >> > > > segments, > >> >> > > > > I > >> >> > > > > > > was > >> >> > > > > > > > > > > > wondering > >> >> > > > > > > > > > > > > > > > whether > >> >> > > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > strictly necessary to do in > the > >> >> > context > >> >> > > > of > >> >> > > > > > this > >> >> > > > > > > > > FLIP > >> >> > > > > > > > > > or > >> >> > > > > > > > > > > > > > whether > >> >> > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > > > > > > > be done as a follow up? > Without > >> >> > knowing > >> >> > > > all > >> >> > > > > > > > > details, > >> >> > > > > > > > > > I > >> >> > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > > concerned > >> >> > > > > > > > > > > > > > > > > > > > > that we would widen the scope > >> of > >> >> this > >> >> > > > FLIP > >> >> > > > > > too > >> >> > > > > > > > much > >> >> > > > > > > > > > > > because > >> >> > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > > > > > > > to touch all the existing > call > >> >> sites > >> >> > of > >> >> > > > the > >> >> > > > > > > > > > > MemoryManager > >> >> > > > > > > > > > > > > > where > >> >> > > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > > > allocate > >> >> > > > > > > > > > > > > > > > > > > > > memory segments (this should > >> >> mainly > >> >> > be > >> >> > > > > batch > >> >> > > > > > > > > > > operators). > >> >> > > > > > > > > > > > > The > >> >> > > > > > > > > > > > > > > > > addition > >> >> > > > > > > > > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > > > > > > > the memory reservation call > to > >> the > >> >> > > > > > > MemoryManager > >> >> > > > > > > > > > should > >> >> > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > affected > >> >> > > > > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > > this and I would hope that > >> this is > >> >> > the > >> >> > > > only > >> >> > > > > > > point > >> >> > > > > > > > > of > >> >> > > > > > > > > > > > > > > interaction > >> >> > > > > > > > > > > > > > > > a > >> >> > > > > > > > > > > > > > > > > > > > > streaming job would have with > >> the > >> >> > > > > > > MemoryManager. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Concerning the second open > >> >> question > >> >> > > about > >> >> > > > > > > setting > >> >> > > > > > > > > or > >> >> > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > setting > >> >> > > > > > > > > > > > > > > > a > >> >> > > > > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > > > > direct memory limit, I would > >> also > >> >> be > >> >> > > > > > interested > >> >> > > > > > > > why > >> >> > > > > > > > > > > Yang > >> >> > > > > > > > > > > > > Wang > >> >> > > > > > > > > > > > > > > > > thinks > >> >> > > > > > > > > > > > > > > > > > > > > leaving it open would be > best. > >> My > >> >> > > concern > >> >> > > > > > about > >> >> > > > > > > > > this > >> >> > > > > > > > > > > > would > >> >> > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > > > > > > > > be in a similar situation as > we > >> >> are > >> >> > now > >> >> > > > > with > >> >> > > > > > > the > >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. > >> >> > > > > > > > > > > > > > > > > > > If > >> >> > > > > > > > > > > > > > > > > > > > > the different memory pools > are > >> not > >> >> > > > clearly > >> >> > > > > > > > > separated > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > can > >> >> > > > > > > > > > > > > > > > spill > >> >> > > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > > a different pool, then it is > >> quite > >> >> > hard > >> >> > > > to > >> >> > > > > > > > > understand > >> >> > > > > > > > > > > > what > >> >> > > > > > > > > > > > > > > > exactly > >> >> > > > > > > > > > > > > > > > > > > > causes a > >> >> > > > > > > > > > > > > > > > > > > > > process to get killed for > using > >> >> too > >> >> > > much > >> >> > > > > > > memory. > >> >> > > > > > > > > This > >> >> > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > then > >> >> > > > > > > > > > > > > > > > > > easily > >> >> > > > > > > > > > > > > > > > > > > > > lead to a similar situation > >> what > >> >> we > >> >> > > have > >> >> > > > > with > >> >> > > > > > > the > >> >> > > > > > > > > > > > > > cutoff-ratio. > >> >> > > > > > > > > > > > > > > > So > >> >> > > > > > > > > > > > > > > > > > why > >> >> > > > > > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > > > setting a sane default value > >> for > >> >> max > >> >> > > > direct > >> >> > > > > > > > memory > >> >> > > > > > > > > > and > >> >> > > > > > > > > > > > > giving > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > an > >> >> > > > > > > > > > > > > > > > > > > > > option to increase it if he > >> runs > >> >> into > >> >> > > an > >> >> > > > > OOM. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, how would > >> alternative 2 > >> >> > lead > >> >> > > to > >> >> > > > > > lower > >> >> > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > utilization > >> >> > > > > > > > > > > > > > > > > > than > >> >> > > > > > > > > > > > > > > > > > > > > alternative 3 where we set > the > >> >> direct > >> >> > > > > memory > >> >> > > > > > > to a > >> >> > > > > > > > > > > higher > >> >> > > > > > > > > > > > > > value? > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 > AM > >> >> > Xintong > >> >> > > > > Song < > >> >> > > > > > > > > > > > > > > > tonysong...@gmail.com > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, > >> Yang. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding your comments: > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* > >> >> > > > > > > > > > > > > > > > > > > > > > I think setting a very > large > >> max > >> >> > > direct > >> >> > > > > > > memory > >> >> > > > > > > > > size > >> >> > > > > > > > > > > > > > > definitely > >> >> > > > > > > > > > > > > > > > > has > >> >> > > > > > > > > > > > > > > > > > > some > >> >> > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not > >> >> worry > >> >> > > about > >> >> > > > > > > direct > >> >> > > > > > > > > OOM, > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > don't > >> >> > > > > > > > > > > > > > > > > > even > >> >> > > > > > > > > > > > > > > > > > > > > need > >> >> > > > > > > > > > > > > > > > > > > > > > to allocate managed / > network > >> >> > memory > >> >> > > > with > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > >> >> > > > > > > > > > > > > > > > > > > > > > However, there are also > some > >> >> down > >> >> > > sides > >> >> > > > > of > >> >> > > > > > > > doing > >> >> > > > > > > > > > > this. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > - One thing I can think > >> of is > >> >> > that > >> >> > > > if > >> >> > > > > a > >> >> > > > > > > task > >> >> > > > > > > > > > > > executor > >> >> > > > > > > > > > > > > > > > > container > >> >> > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > > killed due to overusing > >> >> memory, > >> >> > it > >> >> > > > > could > >> >> > > > > > > be > >> >> > > > > > > > > hard > >> >> > > > > > > > > > > for > >> >> > > > > > > > > > > > > use > >> >> > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > know > >> >> > > > > > > > > > > > > > > > > > > > which > >> >> > > > > > > > > > > > > > > > > > > > > > part > >> >> > > > > > > > > > > > > > > > > > > > > > of the memory is > overused. > >> >> > > > > > > > > > > > > > > > > > > > > > - Another down side is > >> that > >> >> the > >> >> > > JVM > >> >> > > > > > never > >> >> > > > > > > > > > trigger > >> >> > > > > > > > > > > GC > >> >> > > > > > > > > > > > > due > >> >> > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > reaching > >> >> > > > > > > > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > > > > > direct memory limit, > >> because > >> >> the > >> >> > > > limit > >> >> > > > > > is > >> >> > > > > > > > too > >> >> > > > > > > > > > high > >> >> > > > > > > > > > > > to > >> >> > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > reached. > >> >> > > > > > > > > > > > > > > > > > > > That > >> >> > > > > > > > > > > > > > > > > > > > > > means we kind of relay > on > >> >> heap > >> >> > > > memory > >> >> > > > > to > >> >> > > > > > > > > trigger > >> >> > > > > > > > > > > GC > >> >> > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > release > >> >> > > > > > > > > > > > > > > > > > > > direct > >> >> > > > > > > > > > > > > > > > > > > > > > memory. That could be a > >> >> problem > >> >> > in > >> >> > > > > cases > >> >> > > > > > > > where > >> >> > > > > > > > > > we > >> >> > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > more > >> >> > > > > > > > > > > > > > > > > > direct > >> >> > > > > > > > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > > > > usage but not enough > heap > >> >> > activity > >> >> > > > to > >> >> > > > > > > > trigger > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > GC. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you can share your > >> reasons > >> >> > for > >> >> > > > > > > preferring > >> >> > > > > > > > > > > > setting a > >> >> > > > > > > > > > > > > > > very > >> >> > > > > > > > > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > > > > > > > value, > >> >> > > > > > > > > > > > > > > > > > > > > > if there are anything else > I > >> >> > > > overlooked. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* > >> >> > > > > > > > > > > > > > > > > > > > > > If there is any conflict > >> between > >> >> > > > multiple > >> >> > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > > > > explicitly specified, I > >> think we > >> >> > > should > >> >> > > > > > throw > >> >> > > > > > > > an > >> >> > > > > > > > > > > error. > >> >> > > > > > > > > > > > > > > > > > > > > > I think doing checking on > the > >> >> > client > >> >> > > > side > >> >> > > > > > is > >> >> > > > > > > a > >> >> > > > > > > > > good > >> >> > > > > > > > > > > > idea, > >> >> > > > > > > > > > > > > > so > >> >> > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can discover the > >> problem > >> >> > > before > >> >> > > > > > > > submitting > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > Flink > >> >> > > > > > > > > > > > > > > > > > cluster, > >> >> > > > > > > > > > > > > > > > > > > > > which > >> >> > > > > > > > > > > > > > > > > > > > > > is always a good thing. > >> >> > > > > > > > > > > > > > > > > > > > > > But we can not only rely on > >> the > >> >> > > client > >> >> > > > > side > >> >> > > > > > > > > > checking, > >> >> > > > > > > > > > > > > > because > >> >> > > > > > > > > > > > > > > > for > >> >> > > > > > > > > > > > > > > > > > > > > > standalone cluster > >> TaskManagers > >> >> on > >> >> > > > > > different > >> >> > > > > > > > > > machines > >> >> > > > > > > > > > > > may > >> >> > > > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > > > > > > different > >> >> > > > > > > > > > > > > > > > > > > > > > configurations and the > client > >> >> does > >> >> > > see > >> >> > > > > > that. > >> >> > > > > > > > > > > > > > > > > > > > > > What do you think? > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 > >> PM > >> >> Yang > >> >> > > > Wang > >> >> > > > > < > >> >> > > > > > > > > > > > > > > > danrtsey...@gmail.com> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi xintong, > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed > >> >> > proposal. > >> >> > > > > After > >> >> > > > > > > all > >> >> > > > > > > > > the > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > > > > > > > > introduced, it will be > more > >> >> > > powerful > >> >> > > > to > >> >> > > > > > > > control > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > flink > >> >> > > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > >> >> > > > > > > > > > > > > > > > > > > > > > > just have few questions > >> about > >> >> it. > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Native and Direct > >> Memory > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > We do not differentiate > >> user > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > and > >> >> > > > > > > > > > > native > >> >> > > > > > > > > > > > > > > memory. > >> >> > > > > > > > > > > > > > > > > > They > >> >> > > > > > > > > > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > > > > > > > all > >> >> > > > > > > > > > > > > > > > > > > > > > > included in task off-heap > >> >> memory. > >> >> > > > > Right? > >> >> > > > > > > So i > >> >> > > > > > > > > > don’t > >> >> > > > > > > > > > > > > think > >> >> > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > > > > set > >> >> > > > > > > > > > > > > > > > > > > > > > > the > -XX:MaxDirectMemorySize > >> >> > > > properly. I > >> >> > > > > > > > prefer > >> >> > > > > > > > > > > > leaving > >> >> > > > > > > > > > > > > > it a > >> >> > > > > > > > > > > > > > > > > very > >> >> > > > > > > > > > > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > If the sum of and > >> fine-grained > >> >> > > > > > > memory(network > >> >> > > > > > > > > > > memory, > >> >> > > > > > > > > > > > > > > managed > >> >> > > > > > > > > > > > > > > > > > > memory, > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > >> >> > > > > > > > > > > > > > > > > > > > > > > is larger than total > >> process > >> >> > > memory, > >> >> > > > > how > >> >> > > > > > do > >> >> > > > > > > > we > >> >> > > > > > > > > > deal > >> >> > > > > > > > > > > > > with > >> >> > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > > situation? > >> >> > > > > > > > > > > > > > > > > > > > > > Do > >> >> > > > > > > > > > > > > > > > > > > > > > > we need to check the > memory > >> >> > > > > configuration > >> >> > > > > > > in > >> >> > > > > > > > > > > client? > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong Song < > >> >> > > tonysong...@gmail.com> > >> >> > > > > > > > > > 于2019年8月7日周三 > >> >> > > > > > > > > > > > > > > 下午10:14写道： > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > We would like to start > a > >> >> > > discussion > >> >> > > > > > > thread > >> >> > > > > > > > on > >> >> > > > > > > > > > > > > "FLIP-49: > >> >> > > > > > > > > > > > > > > > > Unified > >> >> > > > > > > > > > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > > > > > > > > > > Configuration for > >> >> > > > TaskExecutors"[1], > >> >> > > > > > > where > >> >> > > > > > > > we > >> >> > > > > > > > > > > > > describe > >> >> > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > improve > >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > >> >> > > configurations. > >> >> > > > > The > >> >> > > > > > > > FLIP > >> >> > > > > > > > > > > > document > >> >> > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > mostly > >> >> > > > > > > > > > > > > > > > > > > > based > >> >> > > > > > > > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > > > > > > an > >> >> > > > > > > > > > > > > > > > > > > > > > > > early design "Memory > >> >> Management > >> >> > > and > >> >> > > > > > > > > > Configuration > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > >> >> > > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, > >> >> > > > > > > > > > > > > > > > > > > > > > > > with updates from > >> follow-up > >> >> > > > > discussions > >> >> > > > > > > > both > >> >> > > > > > > > > > > online > >> >> > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > > offline. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses > >> several > >> >> > > > > > shortcomings > >> >> > > > > > > of > >> >> > > > > > > > > > > current > >> >> > > > > > > > > > > > > > > (Flink > >> >> > > > > > > > > > > > > > > > > 1.9) > >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > >> >> > > configuration. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Different > >> configuration > >> >> > for > >> >> > > > > > > Streaming > >> >> > > > > > > > > and > >> >> > > > > > > > > > > > Batch. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complex and > >> difficult > >> >> > > > > > configuration > >> >> > > > > > > of > >> >> > > > > > > > > > > RocksDB > >> >> > > > > > > > > > > > > in > >> >> > > > > > > > > > > > > > > > > > Streaming. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complicated, > >> uncertain > >> >> and > >> >> > > > hard > >> >> > > > > to > >> >> > > > > > > > > > > understand. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve > the > >> >> > problems > >> >> > > > can > >> >> > > > > > be > >> >> > > > > > > > > > > summarized > >> >> > > > > > > > > > > > > as > >> >> > > > > > > > > > > > > > > > > follows. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Extend memory > >> manager > >> >> to > >> >> > > also > >> >> > > > > > > account > >> >> > > > > > > > > for > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > usage > >> >> > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > state > >> >> > > > > > > > > > > > > > > > > > > > > > > > backends. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Modify how > >> TaskExecutor > >> >> > > memory > >> >> > > > > is > >> >> > > > > > > > > > > partitioned > >> >> > > > > > > > > > > > > > > > accounted > >> >> > > > > > > > > > > > > > > > > > > > > individual > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory reservations > >> and > >> >> > pools. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory > >> >> > > configuration > >> >> > > > > > > options > >> >> > > > > > > > > and > >> >> > > > > > > > > > > > > > > calculations > >> >> > > > > > > > > > > > > > > > > > > logics. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please find more > details > >> in > >> >> the > >> >> > > > FLIP > >> >> > > > > > wiki > >> >> > > > > > > > > > > document > >> >> > > > > > > > > > > > > [1]. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > (Please note that the > >> early > >> >> > > design > >> >> > > > > doc > >> >> > > > > > > [2] > >> >> > > > > > > > is > >> >> > > > > > > > > > out > >> >> > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > sync, > >> >> > > > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > > it > >> >> > > > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the > >> >> > > discussion > >> >> > > > in > >> >> > > > > > > this > >> >> > > > > > > > > > > mailing > >> >> > > > > > > > > > > > > list > >> >> > > > > > > > > > > > > > > > > > thread.) > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your > >> >> > > feedbacks. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong > Song > >> < > >> >> > > > > > > > > > tonysong...@gmail.com> > >> >> > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thanks for sharing your opinion Till. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was > >> >> > wondering > >> >> > > > > > whether > >> >> > > > > > > > we > >> >> > > > > > > > > > can > >> >> > > > > > > > > > > > > avoid > >> >> > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap > >> managed > >> >> > memory > >> >> > > > and > >> >> > > > > > > > network > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > with > >> >> > > > > > > > > > > > > > > alternative 3. But after giving it a > second > >> >> > > thought, > >> >> > > > I > >> >> > > > > > > think > >> >> > > > > > > > > even > >> >> > > > > > > > > > > for > >> >> > > > > > > > > > > > > > > alternative 3 using direct memory for > >> off-heap > >> >> > > > managed > >> >> > > > > > > memory > >> >> > > > > > > > > > could > >> >> > > > > > > > > > > > > cause > >> >> > > > > > > > > > > > > > > problems. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Hi Yang, > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Regarding your concern, I think what > >> proposed > >> >> in > >> >> > > this > >> >> > > > > > FLIP > >> >> > > > > > > it > >> >> > > > > > > > > to > >> >> > > > > > > > > > > have > >> >> > > > > > > > > > > > > > both > >> >> > > > > > > > > > > > > > > off-heap managed memory and network > memory > >> >> > > allocated > >> >> > > > > > > through > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which means they are > >> >> > practically > >> >> > > > > > native > >> >> > > > > > > > > memory > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > limited by JVM max direct memory. The > only > >> >> parts > >> >> > of > >> >> > > > > > memory > >> >> > > > > > > > > > limited > >> >> > > > > > > > > > > by > >> >> > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > max direct memory are task off-heap > memory > >> and > >> >> > JVM > >> >> > > > > > > overhead, > >> >> > > > > > > > > > which > >> >> > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > exactly alternative 2 suggests to set the > >> JVM > >> >> max > >> >> > > > > direct > >> >> > > > > > > > memory > >> >> > > > > > > > > > to. > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till > >> Rohrmann > >> >> < > >> >> > > > > > > > > > > trohrm...@apache.org> > >> >> > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I > >> >> > > understand > >> >> > > > > the > >> >> > > > > > > two > >> >> > > > > > > > > > > > > alternatives > >> >> > > > > > > > > > > > > > > > now. > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > I would be in favour of option 2 > because > >> it > >> >> > makes > >> >> > > > > > things > >> >> > > > > > > > > > > explicit. > >> >> > > > > > > > > > > > If > >> >> > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > don't limit the direct memory, I fear > >> that > >> >> we > >> >> > > might > >> >> > > > > end > >> >> > > > > > > up > >> >> > > > > > > > > in a > >> >> > > > > > > > > > > > > similar > >> >> > > > > > > > > > > > > > > > situation as we are currently in: The > >> user > >> >> > might > >> >> > > > see > >> >> > > > > > that > >> >> > > > > > > > her > >> >> > > > > > > > > > > > process > >> >> > > > > > > > > > > > > > > gets > >> >> > > > > > > > > > > > > > > > killed by the OS and does not know why > >> this > >> >> is > >> >> > > the > >> >> > > > > > case. > >> >> > > > > > > > > > > > > Consequently, > >> >> > > > > > > > > > > > > > > she > >> >> > > > > > > > > > > > > > > > tries to decrease the process memory > size > >> >> > > (similar > >> >> > > > to > >> >> > > > > > > > > > increasing > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > cutoff > >> >> > > > > > > > > > > > > > > > ratio) in order to accommodate for the > >> extra > >> >> > > direct > >> >> > > > > > > memory. > >> >> > > > > > > > > > Even > >> >> > > > > > > > > > > > > worse, > >> >> > > > > > > > > > > > > > > she > >> >> > > > > > > > > > > > > > > > tries to decrease memory budgets which > >> are > >> >> not > >> >> > > > fully > >> >> > > > > > used > >> >> > > > > > > > and > >> >> > > > > > > > > > > hence > >> >> > > > > > > > > > > > > > won't > >> >> > > > > > > > > > > > > > > > change the overall memory consumption. > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM > Xintong > >> >> Song < > >> >> > > > > > > > > > > > tonysong...@gmail.com > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let me explain this with a concrete > >> >> example > >> >> > > Till. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let's say we have the following > >> scenario. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Total Process Memory: 1GB > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap > >> Memory + > >> >> JVM > >> >> > > > > > > Overhead): > >> >> > > > > > > > > > 200MB > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM > >> >> Metaspace, > >> >> > > > > > Off-Heap > >> >> > > > > > > > > > Managed > >> >> > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > Network Memory): 800MB > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > For alternative 2, we set > >> >> > > -XX:MaxDirectMemorySize > >> >> > > > > to > >> >> > > > > > > > 200MB. > >> >> > > > > > > > > > > > > > > > > For alternative 3, we set > >> >> > > -XX:MaxDirectMemorySize > >> >> > > > > to > >> >> > > > > > a > >> >> > > > > > > > very > >> >> > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > value, > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of > >> Task > >> >> > > > Off-Heap > >> >> > > > > > > Memory > >> >> > > > > > > > > and > >> >> > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > Overhead > >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, then > alternative 2 > >> >> and > >> >> > > > > > > alternative 3 > >> >> > > > > > > > > > > should > >> >> > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > same utility. Setting larger > >> >> > > > > -XX:MaxDirectMemorySize > >> >> > > > > > > will > >> >> > > > > > > > > not > >> >> > > > > > > > > > > > > reduce > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > sizes of the other memory pools. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of > >> Task > >> >> > > > Off-Heap > >> >> > > > > > > Memory > >> >> > > > > > > > > and > >> >> > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, > then > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > - Alternative 2 suffers from > >> frequent > >> >> OOM. > >> >> > > To > >> >> > > > > > avoid > >> >> > > > > > > > > that, > >> >> > > > > > > > > > > the > >> >> > > > > > > > > > > > > only > >> >> > > > > > > > > > > > > > > > thing > >> >> > > > > > > > > > > > > > > > > user can do is to modify the > >> >> configuration > >> >> > > and > >> >> > > > > > > > increase > >> >> > > > > > > > > > JVM > >> >> > > > > > > > > > > > > Direct > >> >> > > > > > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM > >> Overhead). > >> >> > Let's > >> >> > > > say > >> >> > > > > > > that > >> >> > > > > > > > > user > >> >> > > > > > > > > > > > > > increases > >> >> > > > > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will > >> >> reduce > >> >> > the > >> >> > > > > total > >> >> > > > > > > > size > >> >> > > > > > > > > of > >> >> > > > > > > > > > > > other > >> >> > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > pools to 750MB, given the total > >> process > >> >> > > memory > >> >> > > > > > > remains > >> >> > > > > > > > > > 1GB. > >> >> > > > > > > > > > > > > > > > > - For alternative 3, there is no > >> >> chance of > >> >> > > > > direct > >> >> > > > > > > OOM. > >> >> > > > > > > > > > There > >> >> > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > chances > >> >> > > > > > > > > > > > > > > > > of exceeding the total process > >> memory > >> >> > limit, > >> >> > > > but > >> >> > > > > > > given > >> >> > > > > > > > > > that > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > process > >> >> > > > > > > > > > > > > > > > > may > >> >> > > > > > > > > > > > > > > > > not use up all the reserved native > >> >> memory > >> >> > > > > > (Off-Heap > >> >> > > > > > > > > > Managed > >> >> > > > > > > > > > > > > > Memory, > >> >> > > > > > > > > > > > > > > > > Network > >> >> > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the > >> actual > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > > usage > >> >> > > > > > > > > > is > >> >> > > > > > > > > > > > > > > slightly > >> >> > > > > > > > > > > > > > > > > above > >> >> > > > > > > > > > > > > > > > > yet very close to 200MB, user > >> probably > >> >> do > >> >> > > not > >> >> > > > > need > >> >> > > > > > > to > >> >> > > > > > > > > > change > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > configurations. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Therefore, I think from the user's > >> >> > > perspective, a > >> >> > > > > > > > feasible > >> >> > > > > > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > > > > for alternative 2 may lead to lower > >> >> resource > >> >> > > > > > > utilization > >> >> > > > > > > > > > > compared > >> >> > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > alternative 3. > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till > >> >> > Rohrmann > >> >> > > < > >> >> > > > > > > > > > > > > trohrm...@apache.org > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > I guess you have to help me > >> understand > >> >> the > >> >> > > > > > difference > >> >> > > > > > > > > > between > >> >> > > > > > > > > > > > > > > > > alternative 2 > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to memory under > utilization > >> >> > > Xintong. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: set > >> >> XX:MaxDirectMemorySize > >> >> > > to > >> >> > > > > Task > >> >> > > > > > > > > > Off-Heap > >> >> > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > JVM > >> >> > > > > > > > > > > > > > > > > > Overhead. Then there is the risk > that > >> >> this > >> >> > > size > >> >> > > > > is > >> >> > > > > > > too > >> >> > > > > > > > > low > >> >> > > > > > > > > > > > > > resulting > >> >> > > > > > > > > > > > > > > > in a > >> >> > > > > > > > > > > > > > > > > > lot of garbage collection and > >> >> potentially > >> >> > an > >> >> > > > OOM. > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: set > >> >> XX:MaxDirectMemorySize > >> >> > > to > >> >> > > > > > > > something > >> >> > > > > > > > > > > larger > >> >> > > > > > > > > > > > > > than > >> >> > > > > > > > > > > > > > > > > > alternative 2. This would of course > >> >> reduce > >> >> > > the > >> >> > > > > > sizes > >> >> > > > > > > of > >> >> > > > > > > > > the > >> >> > > > > > > > > > > > other > >> >> > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > types. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > How would alternative 2 now result > >> in an > >> >> > > under > >> >> > > > > > > > > utilization > >> >> > > > > > > > > > of > >> >> > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > compared to alternative 3? If > >> >> alternative 3 > >> >> > > > > > strictly > >> >> > > > > > > > > sets a > >> >> > > > > > > > > > > > > higher > >> >> > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > direct memory size and we use only > >> >> little, > >> >> > > > then I > >> >> > > > > > > would > >> >> > > > > > > > > > > expect > >> >> > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > alternative 3 results in memory > under > >> >> > > > > utilization. > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM > Yang > >> >> Wang < > >> >> > > > > > > > > > > > danrtsey...@gmail.com > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Native and Direct Memory > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > My point is setting a very large > >> max > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > size > >> >> > > > > > > > > > > when > >> >> > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > do > >> >> > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > differentiate direct and native > >> >> memory. > >> >> > If > >> >> > > > the > >> >> > > > > > > direct > >> >> > > > > > > > > > > > > > > > memory,including > >> >> > > > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > direct memory and framework > direct > >> >> > > > memory,could > >> >> > > > > > be > >> >> > > > > > > > > > > calculated > >> >> > > > > > > > > > > > > > > > > > > correctly,then > >> >> > > > > > > > > > > > > > > > > > > i am in favor of setting direct > >> memory > >> >> > with > >> >> > > > > fixed > >> >> > > > > > > > > value. > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Memory Calculation > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn > and > >> >> k8s,we > >> >> > > > need > >> >> > > > > to > >> >> > > > > > > > check > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > configurations in client to avoid > >> >> > > submitting > >> >> > > > > > > > > successfully > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > failing > >> >> > > > > > > > > > > > > > > > > in > >> >> > > > > > > > > > > > > > > > > > > the flink master. > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Best, > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Yang > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > >> tonysong...@gmail.com > >> >> > > > > >于2019年8月13日 > >> >> > > > > > > > > > 周二22:07写道： > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About MemorySegment, I think > you > >> are > >> >> > > right > >> >> > > > > that > >> >> > > > > > > we > >> >> > > > > > > > > > should > >> >> > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > include > >> >> > > > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > issue in the scope of this > FLIP. > >> >> This > >> >> > > FLIP > >> >> > > > > > should > >> >> > > > > > > > > > > > concentrate > >> >> > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > configure memory pools for > >> >> > TaskExecutors, > >> >> > > > > with > >> >> > > > > > > > > minimum > >> >> > > > > > > > > > > > > > > involvement > >> >> > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > > > > > memory consumers use it. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About direct memory, I think > >> >> > alternative > >> >> > > 3 > >> >> > > > > may > >> >> > > > > > > not > >> >> > > > > > > > > > having > >> >> > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > same > >> >> > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > reservation issue that > >> alternative 2 > >> >> > > does, > >> >> > > > > but > >> >> > > > > > at > >> >> > > > > > > > the > >> >> > > > > > > > > > > cost > >> >> > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > risk > >> >> > > > > > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > using memory at the container > >> level, > >> >> > > which > >> >> > > > is > >> >> > > > > > not > >> >> > > > > > > > > good. > >> >> > > > > > > > > > > My > >> >> > > > > > > > > > > > > > point > >> >> > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and > >> "JVM > >> >> > > > > Overhead" > >> >> > > > > > > are > >> >> > > > > > > > > not > >> >> > > > > > > > > > > easy > >> >> > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > config. > >> >> > > > > > > > > > > > > > > > > > > For > >> >> > > > > > > > > > > > > > > > > > > > alternative 2, users might > >> configure > >> >> > them > >> >> > > > > > higher > >> >> > > > > > > > than > >> >> > > > > > > > > > > what > >> >> > > > > > > > > > > > > > > actually > >> >> > > > > > > > > > > > > > > > > > > needed, > >> >> > > > > > > > > > > > > > > > > > > > just to avoid getting a direct > >> OOM. > >> >> For > >> >> > > > > > > alternative > >> >> > > > > > > > > 3, > >> >> > > > > > > > > > > > users > >> >> > > > > > > > > > > > > do > >> >> > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > get > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so they may not > >> config > >> >> the > >> >> > > two > >> >> > > > > > > options > >> >> > > > > > > > > > > > > aggressively > >> >> > > > > > > > > > > > > > > > high. > >> >> > > > > > > > > > > > > > > > > > But > >> >> > > > > > > > > > > > > > > > > > > > the consequences are risks of > >> >> overall > >> >> > > > > container > >> >> > > > > > > > > memory > >> >> > > > > > > > > > > > usage > >> >> > > > > > > > > > > > > > > > exceeds > >> >> > > > > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > > > > budget. > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM > >> Till > >> >> > > > > Rohrmann < > >> >> > > > > > > > > > > > > > > > trohrm...@apache.org> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for proposing this > FLIP > >> >> > Xintong. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > All in all I think it already > >> >> looks > >> >> > > quite > >> >> > > > > > good. > >> >> > > > > > > > > > > > Concerning > >> >> > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > first > >> >> > > > > > > > > > > > > > > > > > > open > >> >> > > > > > > > > > > > > > > > > > > > > question about allocating > >> memory > >> >> > > > segments, > >> >> > > > > I > >> >> > > > > > > was > >> >> > > > > > > > > > > > wondering > >> >> > > > > > > > > > > > > > > > whether > >> >> > > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > strictly necessary to do in > the > >> >> > context > >> >> > > > of > >> >> > > > > > this > >> >> > > > > > > > > FLIP > >> >> > > > > > > > > > or > >> >> > > > > > > > > > > > > > whether > >> >> > > > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > > > > > > > be done as a follow up? > Without > >> >> > knowing > >> >> > > > all > >> >> > > > > > > > > details, > >> >> > > > > > > > > > I > >> >> > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > > concerned > >> >> > > > > > > > > > > > > > > > > > > > > that we would widen the scope > >> of > >> >> this > >> >> > > > FLIP > >> >> > > > > > too > >> >> > > > > > > > much > >> >> > > > > > > > > > > > because > >> >> > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > > > > > > > to touch all the existing > call > >> >> sites > >> >> > of > >> >> > > > the > >> >> > > > > > > > > > > MemoryManager > >> >> > > > > > > > > > > > > > where > >> >> > > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > > > allocate > >> >> > > > > > > > > > > > > > > > > > > > > memory segments (this should > >> >> mainly > >> >> > be > >> >> > > > > batch > >> >> > > > > > > > > > > operators). > >> >> > > > > > > > > > > > > The > >> >> > > > > > > > > > > > > > > > > addition > >> >> > > > > > > > > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > > > > > > > the memory reservation call > to > >> the > >> >> > > > > > > MemoryManager > >> >> > > > > > > > > > should > >> >> > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > affected > >> >> > > > > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > > this and I would hope that > >> this is > >> >> > the > >> >> > > > only > >> >> > > > > > > point > >> >> > > > > > > > > of > >> >> > > > > > > > > > > > > > > interaction > >> >> > > > > > > > > > > > > > > > a > >> >> > > > > > > > > > > > > > > > > > > > > streaming job would have with > >> the > >> >> > > > > > > MemoryManager. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Concerning the second open > >> >> question > >> >> > > about > >> >> > > > > > > setting > >> >> > > > > > > > > or > >> >> > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > setting > >> >> > > > > > > > > > > > > > > > a > >> >> > > > > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > > > > direct memory limit, I would > >> also > >> >> be > >> >> > > > > > interested > >> >> > > > > > > > why > >> >> > > > > > > > > > > Yang > >> >> > > > > > > > > > > > > Wang > >> >> > > > > > > > > > > > > > > > > thinks > >> >> > > > > > > > > > > > > > > > > > > > > leaving it open would be > best. > >> My > >> >> > > concern > >> >> > > > > > about > >> >> > > > > > > > > this > >> >> > > > > > > > > > > > would > >> >> > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > > > would > >> >> > > > > > > > > > > > > > > > > > > > > be in a similar situation as > we > >> >> are > >> >> > now > >> >> > > > > with > >> >> > > > > > > the > >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. > >> >> > > > > > > > > > > > > > > > > > > If > >> >> > > > > > > > > > > > > > > > > > > > > the different memory pools > are > >> not > >> >> > > > clearly > >> >> > > > > > > > > separated > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > can > >> >> > > > > > > > > > > > > > > > spill > >> >> > > > > > > > > > > > > > > > > > over > >> >> > > > > > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > > a different pool, then it is > >> quite > >> >> > hard > >> >> > > > to > >> >> > > > > > > > > understand > >> >> > > > > > > > > > > > what > >> >> > > > > > > > > > > > > > > > exactly > >> >> > > > > > > > > > > > > > > > > > > > causes a > >> >> > > > > > > > > > > > > > > > > > > > > process to get killed for > using > >> >> too > >> >> > > much > >> >> > > > > > > memory. > >> >> > > > > > > > > This > >> >> > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > then > >> >> > > > > > > > > > > > > > > > > > easily > >> >> > > > > > > > > > > > > > > > > > > > > lead to a similar situation > >> what > >> >> we > >> >> > > have > >> >> > > > > with > >> >> > > > > > > the > >> >> > > > > > > > > > > > > > cutoff-ratio. > >> >> > > > > > > > > > > > > > > > So > >> >> > > > > > > > > > > > > > > > > > why > >> >> > > > > > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > > > setting a sane default value > >> for > >> >> max > >> >> > > > direct > >> >> > > > > > > > memory > >> >> > > > > > > > > > and > >> >> > > > > > > > > > > > > giving > >> >> > > > > > > > > > > > > > > the > >> >> > > > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > an > >> >> > > > > > > > > > > > > > > > > > > > > option to increase it if he > >> runs > >> >> into > >> >> > > an > >> >> > > > > OOM. > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, how would > >> alternative 2 > >> >> > lead > >> >> > > to > >> >> > > > > > lower > >> >> > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > utilization > >> >> > > > > > > > > > > > > > > > > > than > >> >> > > > > > > > > > > > > > > > > > > > > alternative 3 where we set > the > >> >> direct > >> >> > > > > memory > >> >> > > > > > > to a > >> >> > > > > > > > > > > higher > >> >> > > > > > > > > > > > > > value? > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > >> >> > > > > > > > > > > > > > > > > > > > > Till > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 > AM > >> >> > Xintong > >> >> > > > > Song < > >> >> > > > > > > > > > > > > > > > tonysong...@gmail.com > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, > >> Yang. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding your comments: > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* > >> >> > > > > > > > > > > > > > > > > > > > > > I think setting a very > large > >> max > >> >> > > direct > >> >> > > > > > > memory > >> >> > > > > > > > > size > >> >> > > > > > > > > > > > > > > definitely > >> >> > > > > > > > > > > > > > > > > has > >> >> > > > > > > > > > > > > > > > > > > some > >> >> > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not > >> >> worry > >> >> > > about > >> >> > > > > > > direct > >> >> > > > > > > > > OOM, > >> >> > > > > > > > > > > and > >> >> > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > don't > >> >> > > > > > > > > > > > > > > > > > even > >> >> > > > > > > > > > > > > > > > > > > > > need > >> >> > > > > > > > > > > > > > > > > > > > > > to allocate managed / > network > >> >> > memory > >> >> > > > with > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > >> >> > > > > > > > > > > > > > > > > > > > > > However, there are also > some > >> >> down > >> >> > > sides > >> >> > > > > of > >> >> > > > > > > > doing > >> >> > > > > > > > > > > this. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > - One thing I can think > >> of is > >> >> > that > >> >> > > > if > >> >> > > > > a > >> >> > > > > > > task > >> >> > > > > > > > > > > > executor > >> >> > > > > > > > > > > > > > > > > container > >> >> > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > > killed due to overusing > >> >> memory, > >> >> > it > >> >> > > > > could > >> >> > > > > > > be > >> >> > > > > > > > > hard > >> >> > > > > > > > > > > for > >> >> > > > > > > > > > > > > use > >> >> > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > know > >> >> > > > > > > > > > > > > > > > > > > > which > >> >> > > > > > > > > > > > > > > > > > > > > > part > >> >> > > > > > > > > > > > > > > > > > > > > > of the memory is > overused. > >> >> > > > > > > > > > > > > > > > > > > > > > - Another down side is > >> that > >> >> the > >> >> > > JVM > >> >> > > > > > never > >> >> > > > > > > > > > trigger > >> >> > > > > > > > > > > GC > >> >> > > > > > > > > > > > > due > >> >> > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > reaching > >> >> > > > > > > > > > > > > > > > > > > > > max > >> >> > > > > > > > > > > > > > > > > > > > > > direct memory limit, > >> because > >> >> the > >> >> > > > limit > >> >> > > > > > is > >> >> > > > > > > > too > >> >> > > > > > > > > > high > >> >> > > > > > > > > > > > to > >> >> > > > > > > > > > > > > be > >> >> > > > > > > > > > > > > > > > > > reached. > >> >> > > > > > > > > > > > > > > > > > > > That > >> >> > > > > > > > > > > > > > > > > > > > > > means we kind of relay > on > >> >> heap > >> >> > > > memory > >> >> > > > > to > >> >> > > > > > > > > trigger > >> >> > > > > > > > > > > GC > >> >> > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > release > >> >> > > > > > > > > > > > > > > > > > > > direct > >> >> > > > > > > > > > > > > > > > > > > > > > memory. That could be a > >> >> problem > >> >> > in > >> >> > > > > cases > >> >> > > > > > > > where > >> >> > > > > > > > > > we > >> >> > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > more > >> >> > > > > > > > > > > > > > > > > > direct > >> >> > > > > > > > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > > > > usage but not enough > heap > >> >> > activity > >> >> > > > to > >> >> > > > > > > > trigger > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > GC. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you can share your > >> reasons > >> >> > for > >> >> > > > > > > preferring > >> >> > > > > > > > > > > > setting a > >> >> > > > > > > > > > > > > > > very > >> >> > > > > > > > > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > > > > > > > value, > >> >> > > > > > > > > > > > > > > > > > > > > > if there are anything else > I > >> >> > > > overlooked. > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* > >> >> > > > > > > > > > > > > > > > > > > > > > If there is any conflict > >> between > >> >> > > > multiple > >> >> > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > user > >> >> > > > > > > > > > > > > > > > > > > > > > explicitly specified, I > >> think we > >> >> > > should > >> >> > > > > > throw > >> >> > > > > > > > an > >> >> > > > > > > > > > > error. > >> >> > > > > > > > > > > > > > > > > > > > > > I think doing checking on > the > >> >> > client > >> >> > > > side > >> >> > > > > > is > >> >> > > > > > > a > >> >> > > > > > > > > good > >> >> > > > > > > > > > > > idea, > >> >> > > > > > > > > > > > > > so > >> >> > > > > > > > > > > > > > > > that > >> >> > > > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can discover the > >> problem > >> >> > > before > >> >> > > > > > > > submitting > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > Flink > >> >> > > > > > > > > > > > > > > > > > cluster, > >> >> > > > > > > > > > > > > > > > > > > > > which > >> >> > > > > > > > > > > > > > > > > > > > > > is always a good thing. > >> >> > > > > > > > > > > > > > > > > > > > > > But we can not only rely on > >> the > >> >> > > client > >> >> > > > > side > >> >> > > > > > > > > > checking, > >> >> > > > > > > > > > > > > > because > >> >> > > > > > > > > > > > > > > > for > >> >> > > > > > > > > > > > > > > > > > > > > > standalone cluster > >> TaskManagers > >> >> on > >> >> > > > > > different > >> >> > > > > > > > > > machines > >> >> > > > > > > > > > > > may > >> >> > > > > > > > > > > > > > > have > >> >> > > > > > > > > > > > > > > > > > > > different > >> >> > > > > > > > > > > > > > > > > > > > > > configurations and the > client > >> >> does > >> >> > > see > >> >> > > > > > that. > >> >> > > > > > > > > > > > > > > > > > > > > > What do you think? > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 > >> PM > >> >> Yang > >> >> > > > Wang > >> >> > > > > < > >> >> > > > > > > > > > > > > > > > danrtsey...@gmail.com> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi xintong, > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed > >> >> > proposal. > >> >> > > > > After > >> >> > > > > > > all > >> >> > > > > > > > > the > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > configuration > >> >> > > > > > > > > > > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > > > > > > > > introduced, it will be > more > >> >> > > powerful > >> >> > > > to > >> >> > > > > > > > control > >> >> > > > > > > > > > the > >> >> > > > > > > > > > > > > flink > >> >> > > > > > > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > >> >> > > > > > > > > > > > > > > > > > > > > > > just have few questions > >> about > >> >> it. > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Native and Direct > >> Memory > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > We do not differentiate > >> user > >> >> > direct > >> >> > > > > > memory > >> >> > > > > > > > and > >> >> > > > > > > > > > > native > >> >> > > > > > > > > > > > > > > memory. > >> >> > > > > > > > > > > > > > > > > > They > >> >> > > > > > > > > > > > > > > > > > > > are > >> >> > > > > > > > > > > > > > > > > > > > > > all > >> >> > > > > > > > > > > > > > > > > > > > > > > included in task off-heap > >> >> memory. > >> >> > > > > Right? > >> >> > > > > > > So i > >> >> > > > > > > > > > don’t > >> >> > > > > > > > > > > > > think > >> >> > > > > > > > > > > > > > > we > >> >> > > > > > > > > > > > > > > > > > could > >> >> > > > > > > > > > > > > > > > > > > > not > >> >> > > > > > > > > > > > > > > > > > > > > > set > >> >> > > > > > > > > > > > > > > > > > > > > > > the > -XX:MaxDirectMemorySize > >> >> > > > properly. I > >> >> > > > > > > > prefer > >> >> > > > > > > > > > > > leaving > >> >> > > > > > > > > > > > > > it a > >> >> > > > > > > > > > > > > > > > > very > >> >> > > > > > > > > > > > > > > > > > > > large > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > If the sum of and > >> fine-grained > >> >> > > > > > > memory(network > >> >> > > > > > > > > > > memory, > >> >> > > > > > > > > > > > > > > managed > >> >> > > > > > > > > > > > > > > > > > > memory, > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > >> >> > > > > > > > > > > > > > > > > > > > > > > is larger than total > >> process > >> >> > > memory, > >> >> > > > > how > >> >> > > > > > do > >> >> > > > > > > > we > >> >> > > > > > > > > > deal > >> >> > > > > > > > > > > > > with > >> >> > > > > > > > > > > > > > > this > >> >> > > > > > > > > > > > > > > > > > > > > situation? > >> >> > > > > > > > > > > > > > > > > > > > > > Do > >> >> > > > > > > > > > > > > > > > > > > > > > > we need to check the > memory > >> >> > > > > configuration > >> >> > > > > > > in > >> >> > > > > > > > > > > client? > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong Song < > >> >> > > tonysong...@gmail.com> > >> >> > > > > > > > > > 于2019年8月7日周三 > >> >> > > > > > > > > > > > > > > 下午10:14写道： > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > We would like to start > a > >> >> > > discussion > >> >> > > > > > > thread > >> >> > > > > > > > on > >> >> > > > > > > > > > > > > "FLIP-49: > >> >> > > > > > > > > > > > > > > > > Unified > >> >> > > > > > > > > > > > > > > > > > > > > Memory > >> >> > > > > > > > > > > > > > > > > > > > > > > > Configuration for > >> >> > > > TaskExecutors"[1], > >> >> > > > > > > where > >> >> > > > > > > > we > >> >> > > > > > > > > > > > > describe > >> >> > > > > > > > > > > > > > > how > >> >> > > > > > > > > > > > > > > > to > >> >> > > > > > > > > > > > > > > > > > > > improve > >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > >> >> > > configurations. > >> >> > > > > The > >> >> > > > > > > > FLIP > >> >> > > > > > > > > > > > document > >> >> > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > mostly > >> >> > > > > > > > > > > > > > > > > > > > based > >> >> > > > > > > > > > > > > > > > > > > > > > on > >> >> > > > > > > > > > > > > > > > > > > > > > > an > >> >> > > > > > > > > > > > > > > > > > > > > > > > early design "Memory > >> >> Management > >> >> > > and > >> >> > > > > > > > > > Configuration > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > >> >> > > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, > >> >> > > > > > > > > > > > > > > > > > > > > > > > with updates from > >> follow-up > >> >> > > > > discussions > >> >> > > > > > > > both > >> >> > > > > > > > > > > online > >> >> > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > > offline. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses > >> several > >> >> > > > > > shortcomings > >> >> > > > > > > of > >> >> > > > > > > > > > > current > >> >> > > > > > > > > > > > > > > (Flink > >> >> > > > > > > > > > > > > > > > > 1.9) > >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > >> >> > > configuration. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Different > >> configuration > >> >> > for > >> >> > > > > > > Streaming > >> >> > > > > > > > > and > >> >> > > > > > > > > > > > Batch. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complex and > >> difficult > >> >> > > > > > configuration > >> >> > > > > > > of > >> >> > > > > > > > > > > RocksDB > >> >> > > > > > > > > > > > > in > >> >> > > > > > > > > > > > > > > > > > Streaming. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complicated, > >> uncertain > >> >> and > >> >> > > > hard > >> >> > > > > to > >> >> > > > > > > > > > > understand. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve > the > >> >> > problems > >> >> > > > can > >> >> > > > > > be > >> >> > > > > > > > > > > summarized > >> >> > > > > > > > > > > > > as > >> >> > > > > > > > > > > > > > > > > follows. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Extend memory > >> manager > >> >> to > >> >> > > also > >> >> > > > > > > account > >> >> > > > > > > > > for > >> >> > > > > > > > > > > > memory > >> >> > > > > > > > > > > > > > > usage > >> >> > > > > > > > > > > > > > > > > by > >> >> > > > > > > > > > > > > > > > > > > > state > >> >> > > > > > > > > > > > > > > > > > > > > > > > backends. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Modify how > >> TaskExecutor > >> >> > > memory > >> >> > > > > is > >> >> > > > > > > > > > > partitioned > >> >> > > > > > > > > > > > > > > > accounted > >> >> > > > > > > > > > > > > > > > > > > > > individual > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory reservations > >> and > >> >> > pools. > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory > >> >> > > configuration > >> >> > > > > > > options > >> >> > > > > > > > > and > >> >> > > > > > > > > > > > > > > calculations > >> >> > > > > > > > > > > > > > > > > > > logics. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please find more > details > >> in > >> >> the > >> >> > > > FLIP > >> >> > > > > > wiki > >> >> > > > > > > > > > > document > >> >> > > > > > > > > > > > > [1]. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > (Please note that the > >> early > >> >> > > design > >> >> > > > > doc > >> >> > > > > > > [2] > >> >> > > > > > > > is > >> >> > > > > > > > > > out > >> >> > > > > > > > > > > > of > >> >> > > > > > > > > > > > > > > sync, > >> >> > > > > > > > > > > > > > > > > and > >> >> > > > > > > > > > > > > > > > > > it > >> >> > > > > > > > > > > > > > > > > > > > is > >> >> > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the > >> >> > > discussion > >> >> > > > in > >> >> > > > > > > this > >> >> > > > > > > > > > > mailing > >> >> > > > > > > > > > > > > list > >> >> > > > > > > > > > > > > > > > > > thread.) > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your > >> >> > > feedbacks. > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > > >> > > >
