I just updated the FLIP wiki page [1], with the following changes: - Network memory uses JVM direct memory, and is accounted when setting JVM max direct memory size parameter. - Use dynamic configurations (`-Dkey=value`) to pass calculated memory configs into TaskExecutors, instead of ENV variables. - Remove 'supporting memory reservation' from the scope of this FLIP.
@till @stephan, please take another look see if there are any other concerns. Thank you~ Xintong Song [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors On Mon, Sep 2, 2019 at 11:13 AM Xintong Song <tonysong...@gmail.com> wrote: > Sorry for the late response. > > - Regarding the `TaskExecutorSpecifics` naming, let's discuss the detail > in PR. > - Regarding passing parameters into the `TaskExecutor`, +1 for using > dynamic configuration at the moment, given that there are more questions to > be discussed to have a general framework for overwriting configurations > with ENV variables. > - Regarding memory reservation, I double checked with Yu and he will take > care of it. > > Thank you~ > > Xintong Song > > > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann <trohrm...@apache.org> > wrote: > >> What I forgot to add is that we could tackle specifying the configuration >> fully in an incremental way and that the full specification should be the >> desired end state. >> >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> > I think our goal should be that the configuration is fully specified >> when >> > the process is started. By considering the internal calculation step to >> be >> > rather validate existing values and calculate missing ones, these two >> > proposal shouldn't even conflict (given determinism). >> > >> > Since we don't want to change an existing flink-conf.yaml, specifying >> the >> > full configuration would require to pass in the options differently. >> > >> > One way could be the ENV variables approach. The reason why I'm trying >> to >> > exclude this feature from the FLIP is that I believe it needs a bit more >> > discussion. Just some questions which come to my mind: What would be the >> > exact format (FLINK_KEY_NAME)? Would we support a dot separator which is >> > supported by some systems (FLINK.KEY.NAME)? If we accept the dot >> > separator what would be the order of precedence if there are two ENV >> > variables defined (FLINK_KEY_NAME and FLINK.KEY.NAME)? What is the >> > precedence of env variable vs. dynamic configuration value specified >> via -D? >> > >> > Another approach could be to pass in the dynamic configuration values >> via >> > `-Dkey=value` to the Flink process. For that we don't have to change >> > anything because the functionality already exists. >> > >> > Cheers, >> > Till >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen <se...@apache.org> wrote: >> > >> >> I see. Under the assumption of strict determinism that should work. >> >> >> >> The original proposal had this point "don't compute inside the TM, >> compute >> >> outside and supply a full config", because that sounded more intuitive. >> >> >> >> On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann <trohrm...@apache.org> >> >> wrote: >> >> >> >> > My understanding was that before starting the Flink process we call a >> >> > utility which calculates these values. I assume that this utility >> will >> >> do >> >> > the calculation based on a set of configured values (process memory, >> >> flink >> >> > memory, network memory etc.). Assuming that these values don't differ >> >> from >> >> > the values with which the JVM is started, it should be possible to >> >> > recompute them in the Flink process in order to set the values. >> >> > >> >> > >> >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen <se...@apache.org> >> wrote: >> >> > >> >> > > When computing the values in the JVM process after it started, how >> >> would >> >> > > you deal with values like Max Direct Memory, Metaspace size. native >> >> > memory >> >> > > reservation (reduce heap size), etc? All the values that are >> >> parameters >> >> > to >> >> > > the JVM process and that need to be supplied at process startup? >> >> > > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann < >> trohrm...@apache.org> >> >> > > wrote: >> >> > > >> >> > > > Thanks for the clarification. I have some more comments: >> >> > > > >> >> > > > - I would actually split the logic to compute the process memory >> >> > > > requirements and storing the values into two things. E.g. one >> could >> >> > name >> >> > > > the former TaskExecutorProcessUtility and the latter >> >> > > > TaskExecutorProcessMemory. But we can discuss this on the PR >> since >> >> it's >> >> > > > just a naming detail. >> >> > > > >> >> > > > - Generally, I'm not opposed to making configuration values >> >> overridable >> >> > > by >> >> > > > ENV variables. I think this is a very good idea and makes the >> >> > > > configurability of Flink processes easier. However, I think that >> >> adding >> >> > > > this functionality should not be part of this FLIP because it >> would >> >> > > simply >> >> > > > widen the scope unnecessarily. >> >> > > > >> >> > > > The reasons why I believe it is unnecessary are the following: >> For >> >> Yarn >> >> > > we >> >> > > > already create write a flink-conf.yaml which could be populated >> with >> >> > the >> >> > > > memory settings. For the other processes it should not make a >> >> > difference >> >> > > > whether the loaded Configuration is populated with the memory >> >> settings >> >> > > from >> >> > > > ENV variables or by using TaskExecutorProcessUtility to compute >> the >> >> > > missing >> >> > > > values from the loaded configuration. If the latter would not be >> >> > possible >> >> > > > (wrong or missing configuration values), then we should not have >> >> been >> >> > > able >> >> > > > to actually start the process in the first place. >> >> > > > >> >> > > > - Concerning the memory reservation: I agree with you that we >> need >> >> the >> >> > > > memory reservation functionality to make streaming jobs work with >> >> > > "managed" >> >> > > > memory. However, w/o this functionality the whole Flip would >> already >> >> > > bring >> >> > > > a good amount of improvements to our users when running batch >> jobs. >> >> > > > Moreover, by keeping the scope smaller we can complete the FLIP >> >> faster. >> >> > > > Hence, I would propose to address the memory reservation >> >> functionality >> >> > > as a >> >> > > > follow up FLIP (which Yu is working on if I'm not mistaken). >> >> > > > >> >> > > > Cheers, >> >> > > > Till >> >> > > > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang < >> danrtsey...@gmail.com> >> >> > > wrote: >> >> > > > >> >> > > > > Just add my 2 cents. >> >> > > > > >> >> > > > > Using environment variables to override the configuration for >> >> > different >> >> > > > > taskmanagers is better. >> >> > > > > We do not need to generate dedicated flink-conf.yaml for all >> >> > > > taskmanagers. >> >> > > > > A common flink-conf.yam and different environment variables are >> >> > enough. >> >> > > > > By reducing the distributed cached files, it could make >> launching >> >> a >> >> > > > > taskmanager faster. >> >> > > > > >> >> > > > > Stephan gives a good suggestion that we could move the logic >> into >> >> > > > > "GlobalConfiguration.loadConfig()" method. >> >> > > > > Maybe the client could also benefit from this. Different users >> do >> >> not >> >> > > > have >> >> > > > > to export FLINK_CONF_DIR to update few config options. >> >> > > > > >> >> > > > > >> >> > > > > Best, >> >> > > > > Yang >> >> > > > > >> >> > > > > Stephan Ewen <se...@apache.org> 于2019年8月28日周三 上午1:21写道： >> >> > > > > >> >> > > > > > One note on the Environment Variables and Configuration >> >> discussion. >> >> > > > > > >> >> > > > > > My understanding is that passed ENV variables are added to >> the >> >> > > > > > configuration in the "GlobalConfiguration.loadConfig()" >> method >> >> (or >> >> > > > > > similar). >> >> > > > > > For all the code inside Flink, it looks like the data was in >> the >> >> > > config >> >> > > > > to >> >> > > > > > start with, just that the scripts that compute the variables >> can >> >> > pass >> >> > > > the >> >> > > > > > values to the process without actually needing to write a >> file. >> >> > > > > > >> >> > > > > > For example the "GlobalConfiguration.loadConfig()" method >> would >> >> > take >> >> > > > any >> >> > > > > > ENV variable prefixed with "flink" and add it as a config >> key. >> >> > > > > > "flink_taskmanager_memory_size=2g" would become >> >> > > > "taskmanager.memory.size: >> >> > > > > > 2g". >> >> > > > > > >> >> > > > > > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM Xintong Song < >> >> > tonysong...@gmail.com> >> >> > > > > > wrote: >> >> > > > > > >> >> > > > > > > Thanks for the comments, Till. >> >> > > > > > > >> >> > > > > > > I've also seen your comments on the wiki page, but let's >> keep >> >> the >> >> > > > > > > discussion here. >> >> > > > > > > >> >> > > > > > > - Regarding 'TaskExecutorSpecifics', how do you think about >> >> > naming >> >> > > it >> >> > > > > > > 'TaskExecutorResourceSpecifics'. >> >> > > > > > > - Regarding passing memory configurations into task >> executors, >> >> > I'm >> >> > > in >> >> > > > > > favor >> >> > > > > > > of do it via environment variables rather than >> configurations, >> >> > with >> >> > > > the >> >> > > > > > > following two reasons. >> >> > > > > > > - It is easier to keep the memory options once calculate >> >> not to >> >> > > be >> >> > > > > > > changed with environment variables rather than >> configurations. >> >> > > > > > > - I'm not sure whether we should write the configuration >> in >> >> > > startup >> >> > > > > > > scripts. Writing changes into the configuration files when >> >> > running >> >> > > > the >> >> > > > > > > startup scripts does not sounds right to me. Or we could >> make >> >> a >> >> > > copy >> >> > > > of >> >> > > > > > > configuration files per flink cluster, and make the task >> >> executor >> >> > > to >> >> > > > > load >> >> > > > > > > from the copy, and clean up the copy after the cluster is >> >> > shutdown, >> >> > > > > which >> >> > > > > > > is complicated. (I think this is also what Stephan means in >> >> his >> >> > > > comment >> >> > > > > > on >> >> > > > > > > the wiki page?) >> >> > > > > > > - Regarding reserving memory, I think this change should be >> >> > > included >> >> > > > in >> >> > > > > > > this FLIP. I think a big part of motivations of this FLIP >> is >> >> to >> >> > > unify >> >> > > > > > > memory configuration for streaming / batch and make it easy >> >> for >> >> > > > > > configuring >> >> > > > > > > rocksdb memory. If we don't support memory reservation, >> then >> >> > > > streaming >> >> > > > > > jobs >> >> > > > > > > cannot use managed memory (neither on-heap or off-heap), >> which >> >> > > makes >> >> > > > > this >> >> > > > > > > FLIP incomplete. >> >> > > > > > > - Regarding network memory, I think you are right. I think >> we >> >> > > > probably >> >> > > > > > > don't need to change network stack from using direct >> memory to >> >> > > using >> >> > > > > > unsafe >> >> > > > > > > native memory. Network memory size is deterministic, >> cannot be >> >> > > > reserved >> >> > > > > > as >> >> > > > > > > managed memory does, and cannot be overused. I think it >> also >> >> > works >> >> > > if >> >> > > > > we >> >> > > > > > > simply keep using direct memory for network and include it >> in >> >> jvm >> >> > > max >> >> > > > > > > direct memory size. >> >> > > > > > > >> >> > > > > > > Thank you~ >> >> > > > > > > >> >> > > > > > > Xintong Song >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann < >> >> > > trohrm...@apache.org> >> >> > > > > > > wrote: >> >> > > > > > > >> >> > > > > > > > Hi Xintong, >> >> > > > > > > > >> >> > > > > > > > thanks for addressing the comments and adding a more >> >> detailed >> >> > > > > > > > implementation plan. I have a couple of comments >> concerning >> >> the >> >> > > > > > > > implementation plan: >> >> > > > > > > > >> >> > > > > > > > - The name `TaskExecutorSpecifics` is not really >> >> descriptive. >> >> > > > > Choosing >> >> > > > > > a >> >> > > > > > > > different name could help here. >> >> > > > > > > > - I'm not sure whether I would pass the memory >> >> configuration to >> >> > > the >> >> > > > > > > > TaskExecutor via environment variables. I think it would >> be >> >> > > better >> >> > > > to >> >> > > > > > > write >> >> > > > > > > > it into the configuration one uses to start the TM >> process. >> >> > > > > > > > - If possible, I would exclude the memory reservation >> from >> >> this >> >> > > > FLIP >> >> > > > > > and >> >> > > > > > > > add this as part of a dedicated FLIP. >> >> > > > > > > > - If possible, then I would exclude changes to the >> network >> >> > stack >> >> > > > from >> >> > > > > > > this >> >> > > > > > > > FLIP. Maybe we can simply say that the direct memory >> needed >> >> by >> >> > > the >> >> > > > > > > network >> >> > > > > > > > stack is the framework direct memory requirement. >> Changing >> >> how >> >> > > the >> >> > > > > > memory >> >> > > > > > > > is allocated can happen in a second step. This would keep >> >> the >> >> > > scope >> >> > > > > of >> >> > > > > > > this >> >> > > > > > > > FLIP smaller. >> >> > > > > > > > >> >> > > > > > > > Cheers, >> >> > > > > > > > Till >> >> > > > > > > > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM Xintong Song < >> >> > > > tonysong...@gmail.com> >> >> > > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Hi everyone, >> >> > > > > > > > > >> >> > > > > > > > > I just updated the FLIP document on wiki [1], with the >> >> > > following >> >> > > > > > > changes. >> >> > > > > > > > > >> >> > > > > > > > > - Removed open question regarding MemorySegment >> >> > allocation. >> >> > > As >> >> > > > > > > > > discussed, we exclude this topic from the scope of >> this >> >> > > FLIP. >> >> > > > > > > > > - Updated content about JVM direct memory parameter >> >> > > according >> >> > > > to >> >> > > > > > > > recent >> >> > > > > > > > > discussions, and moved the other options to >> "Rejected >> >> > > > > > Alternatives" >> >> > > > > > > > for >> >> > > > > > > > > the >> >> > > > > > > > > moment. >> >> > > > > > > > > - Added implementation steps. >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > Thank you~ >> >> > > > > > > > > >> >> > > > > > > > > Xintong Song >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > [1] >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >> >> > > > > > > > > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM Stephan Ewen < >> >> > se...@apache.org >> >> > > > >> >> > > > > > wrote: >> >> > > > > > > > > >> >> > > > > > > > > > @Xintong: Concerning "wait for memory users before >> task >> >> > > dispose >> >> > > > > and >> >> > > > > > > > > memory >> >> > > > > > > > > > release": I agree, that's how it should be. Let's >> try it >> >> > out. >> >> > > > > > > > > > >> >> > > > > > > > > > @Xintong @Jingsong: Concerning " JVM does not wait >> for >> >> GC >> >> > > when >> >> > > > > > > > allocating >> >> > > > > > > > > > direct memory buffer": There seems to be pretty >> >> elaborate >> >> > > logic >> >> > > > > to >> >> > > > > > > free >> >> > > > > > > > > > buffers when allocating new ones. See >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 >> >> > > > > > > > > > >> >> > > > > > > > > > @Till: Maybe. If we assume that the JVM default works >> >> (like >> >> > > > going >> >> > > > > > > with >> >> > > > > > > > > > option 2 and not setting "-XX:MaxDirectMemorySize" at >> >> all), >> >> > > > then >> >> > > > > I >> >> > > > > > > > think >> >> > > > > > > > > it >> >> > > > > > > > > > should be okay to set "-XX:MaxDirectMemorySize" to >> >> > > > > > > > > > "off_heap_managed_memory + direct_memory" even if we >> use >> >> > > > RocksDB. >> >> > > > > > > That >> >> > > > > > > > > is a >> >> > > > > > > > > > big if, though, I honestly have no idea :D Would be >> >> good to >> >> > > > > > > understand >> >> > > > > > > > > > this, though, because this would affect option (2) >> and >> >> > option >> >> > > > > > (1.2). >> >> > > > > > > > > > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 PM Xintong Song < >> >> > > > > > tonysong...@gmail.com> >> >> > > > > > > > > > wrote: >> >> > > > > > > > > > >> >> > > > > > > > > > > Thanks for the inputs, Jingsong. >> >> > > > > > > > > > > >> >> > > > > > > > > > > Let me try to summarize your points. Please correct >> >> me if >> >> > > I'm >> >> > > > > > > wrong. >> >> > > > > > > > > > > >> >> > > > > > > > > > > - Memory consumers should always avoid returning >> >> > memory >> >> > > > > > segments >> >> > > > > > > > to >> >> > > > > > > > > > > memory manager while there are still un-cleaned >> >> > > > structures / >> >> > > > > > > > threads >> >> > > > > > > > > > > that >> >> > > > > > > > > > > may use the memory. Otherwise, it would cause >> >> serious >> >> > > > > problems >> >> > > > > > > by >> >> > > > > > > > > > having >> >> > > > > > > > > > > multiple consumers trying to use the same memory >> >> > > segment. >> >> > > > > > > > > > > - JVM does not wait for GC when allocating >> direct >> >> > memory >> >> > > > > > buffer. >> >> > > > > > > > > > > Therefore even we set proper max direct memory >> size >> >> > > limit, >> >> > > > > we >> >> > > > > > > may >> >> > > > > > > > > > still >> >> > > > > > > > > > > encounter direct memory oom if the GC cleaning >> >> memory >> >> > > > slower >> >> > > > > > > than >> >> > > > > > > > > the >> >> > > > > > > > > > > direct memory allocation. >> >> > > > > > > > > > > >> >> > > > > > > > > > > Am I understanding this correctly? >> >> > > > > > > > > > > >> >> > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > >> >> > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 PM JingsongLee < >> >> > > > > > > lzljs3620...@aliyun.com >> >> > > > > > > > > > > .invalid> >> >> > > > > > > > > > > wrote: >> >> > > > > > > > > > > >> >> > > > > > > > > > > > Hi stephan: >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > About option 2: >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > if additional threads not cleanly shut down >> before >> >> we >> >> > can >> >> > > > > exit >> >> > > > > > > the >> >> > > > > > > > > > task: >> >> > > > > > > > > > > > In the current case of memory reuse, it has >> freed up >> >> > the >> >> > > > > memory >> >> > > > > > > it >> >> > > > > > > > > > > > uses. If this memory is used by other tasks and >> >> > > > asynchronous >> >> > > > > > > > threads >> >> > > > > > > > > > > > of exited task may still be writing, there will >> be >> >> > > > > concurrent >> >> > > > > > > > > security >> >> > > > > > > > > > > > problems, and even lead to errors in user >> computing >> >> > > > results. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > So I think this is a serious and intolerable >> bug, No >> >> > > matter >> >> > > > > > what >> >> > > > > > > > the >> >> > > > > > > > > > > > option is, it should be avoided. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > About direct memory cleaned by GC: >> >> > > > > > > > > > > > I don't think it is a good idea, I've >> encountered so >> >> > many >> >> > > > > > > > situations >> >> > > > > > > > > > > > that it's too late for GC to cause DirectMemory >> >> OOM. >> >> > > > Release >> >> > > > > > and >> >> > > > > > > > > > > > allocate DirectMemory depend on the type of user >> >> job, >> >> > > > which >> >> > > > > is >> >> > > > > > > > > > > > often beyond our control. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > Best, >> >> > > > > > > > > > > > Jingsong Lee >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > ------------------------------------------------------------------ >> >> > > > > > > > > > > > From:Stephan Ewen <se...@apache.org> >> >> > > > > > > > > > > > Send Time:2019年8月19日(星期一) 15:56 >> >> > > > > > > > > > > > To:dev <dev@flink.apache.org> >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] FLIP-49: Unified Memory >> >> > > Configuration >> >> > > > > for >> >> > > > > > > > > > > > TaskExecutors >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > My main concern with option 2 (manually release >> >> memory) >> >> > > is >> >> > > > > that >> >> > > > > > > > > > segfaults >> >> > > > > > > > > > > > in the JVM send off all sorts of alarms on user >> >> ends. >> >> > So >> >> > > we >> >> > > > > > need >> >> > > > > > > to >> >> > > > > > > > > > > > guarantee that this never happens. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > The trickyness is in tasks that uses data >> >> structures / >> >> > > > > > algorithms >> >> > > > > > > > > with >> >> > > > > > > > > > > > additional threads, like hash table spill/read >> and >> >> > > sorting >> >> > > > > > > threads. >> >> > > > > > > > > We >> >> > > > > > > > > > > need >> >> > > > > > > > > > > > to ensure that these cleanly shut down before we >> can >> >> > exit >> >> > > > the >> >> > > > > > > task. >> >> > > > > > > > > > > > I am not sure that we have that guaranteed >> already, >> >> > > that's >> >> > > > > why >> >> > > > > > > > option >> >> > > > > > > > > > 1.1 >> >> > > > > > > > > > > > seemed simpler to me. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at 3:42 PM Xintong Song < >> >> > > > > > > > tonysong...@gmail.com> >> >> > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > > Thanks for the comments, Stephan. Summarized in >> >> this >> >> > > way >> >> > > > > > really >> >> > > > > > > > > makes >> >> > > > > > > > > > > > > things easier to understand. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > I'm in favor of option 2, at least for the >> >> moment. I >> >> > > > think >> >> > > > > it >> >> > > > > > > is >> >> > > > > > > > > not >> >> > > > > > > > > > > that >> >> > > > > > > > > > > > > difficult to keep it segfault safe for memory >> >> > manager, >> >> > > as >> >> > > > > > long >> >> > > > > > > as >> >> > > > > > > > > we >> >> > > > > > > > > > > > always >> >> > > > > > > > > > > > > de-allocate the memory segment when it is >> released >> >> > from >> >> > > > the >> >> > > > > > > > memory >> >> > > > > > > > > > > > > consumers. Only if the memory consumer continue >> >> using >> >> > > the >> >> > > > > > > buffer >> >> > > > > > > > of >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > segment after releasing it, in which case we do >> >> want >> >> > > the >> >> > > > > job >> >> > > > > > to >> >> > > > > > > > > fail >> >> > > > > > > > > > so >> >> > > > > > > > > > > > we >> >> > > > > > > > > > > > > detect the memory leak early. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > For option 1.2, I don't think this is a good >> idea. >> >> > Not >> >> > > > only >> >> > > > > > > > because >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > assumption (regular GC is enough to clean >> direct >> >> > > buffers) >> >> > > > > may >> >> > > > > > > not >> >> > > > > > > > > > > always >> >> > > > > > > > > > > > be >> >> > > > > > > > > > > > > true, but also it makes harder for finding >> >> problems >> >> > in >> >> > > > > cases >> >> > > > > > of >> >> > > > > > > > > > memory >> >> > > > > > > > > > > > > overuse. E.g., user configured some direct >> memory >> >> for >> >> > > the >> >> > > > > > user >> >> > > > > > > > > > > libraries. >> >> > > > > > > > > > > > > If the library actually use more direct memory >> >> then >> >> > > > > > configured, >> >> > > > > > > > > which >> >> > > > > > > > > > > > > cannot be cleaned by GC because they are still >> in >> >> > use, >> >> > > > may >> >> > > > > > lead >> >> > > > > > > > to >> >> > > > > > > > > > > > overuse >> >> > > > > > > > > > > > > of the total container memory. In that case, >> if it >> >> > > didn't >> >> > > > > > touch >> >> > > > > > > > the >> >> > > > > > > > > > JVM >> >> > > > > > > > > > > > > default max direct memory limit, we cannot get >> a >> >> > direct >> >> > > > > > memory >> >> > > > > > > > OOM >> >> > > > > > > > > > and >> >> > > > > > > > > > > it >> >> > > > > > > > > > > > > will become super hard to understand which >> part of >> >> > the >> >> > > > > > > > > configuration >> >> > > > > > > > > > > need >> >> > > > > > > > > > > > > to be updated. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > For option 1.1, it has the similar problem as >> >> 1.2, if >> >> > > the >> >> > > > > > > > exceeded >> >> > > > > > > > > > > direct >> >> > > > > > > > > > > > > memory does not reach the max direct memory >> limit >> >> > > > specified >> >> > > > > > by >> >> > > > > > > > the >> >> > > > > > > > > > > > > dedicated parameter. I think it is slightly >> better >> >> > than >> >> > > > > 1.2, >> >> > > > > > > only >> >> > > > > > > > > > > because >> >> > > > > > > > > > > > > we can tune the parameter. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen < >> >> > > > > > se...@apache.org >> >> > > > > > > > >> >> > > > > > > > > > wrote: >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > > About the "-XX:MaxDirectMemorySize" >> discussion, >> >> > maybe >> >> > > > let >> >> > > > > > me >> >> > > > > > > > > > > summarize >> >> > > > > > > > > > > > > it a >> >> > > > > > > > > > > > > > bit differently: >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > We have the following two options: >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > (1) We let MemorySegments be de-allocated by >> the >> >> > GC. >> >> > > > That >> >> > > > > > > makes >> >> > > > > > > > > it >> >> > > > > > > > > > > > > segfault >> >> > > > > > > > > > > > > > safe. But then we need a way to trigger GC in >> >> case >> >> > > > > > > > de-allocation >> >> > > > > > > > > > and >> >> > > > > > > > > > > > > > re-allocation of a bunch of segments happens >> >> > quickly, >> >> > > > > which >> >> > > > > > > is >> >> > > > > > > > > > often >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > case during batch scheduling or task restart. >> >> > > > > > > > > > > > > > - The "-XX:MaxDirectMemorySize" (option >> 1.1) >> >> is >> >> > one >> >> > > > way >> >> > > > > > to >> >> > > > > > > do >> >> > > > > > > > > > this >> >> > > > > > > > > > > > > > - Another way could be to have a dedicated >> >> > > > bookkeeping >> >> > > > > in >> >> > > > > > > the >> >> > > > > > > > > > > > > > MemoryManager (option 1.2), so that this is a >> >> > number >> >> > > > > > > > independent >> >> > > > > > > > > of >> >> > > > > > > > > > > the >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" parameter. >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > (2) We manually allocate and de-allocate the >> >> memory >> >> > > for >> >> > > > > the >> >> > > > > > > > > > > > > MemorySegments >> >> > > > > > > > > > > > > > (option 2). That way we need not worry about >> >> > > triggering >> >> > > > > GC >> >> > > > > > by >> >> > > > > > > > > some >> >> > > > > > > > > > > > > > threshold or bookkeeping, but it is harder to >> >> > prevent >> >> > > > > > > > segfaults. >> >> > > > > > > > > We >> >> > > > > > > > > > > > need >> >> > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > be very careful about when we release the >> memory >> >> > > > segments >> >> > > > > > > (only >> >> > > > > > > > > in >> >> > > > > > > > > > > the >> >> > > > > > > > > > > > > > cleanup phase of the main thread). >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > If we go with option 1.1, we probably need to >> >> set >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" to >> >> > > "off_heap_managed_memory + >> >> > > > > > > > > > > direct_memory" >> >> > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > have "direct_memory" as a separate reserved >> >> memory >> >> > > > pool. >> >> > > > > > > > Because >> >> > > > > > > > > if >> >> > > > > > > > > > > we >> >> > > > > > > > > > > > > just >> >> > > > > > > > > > > > > > set "-XX:MaxDirectMemorySize" to >> >> > > > > "off_heap_managed_memory + >> >> > > > > > > > > > > > > jvm_overhead", >> >> > > > > > > > > > > > > > then there will be times when that entire >> >> memory is >> >> > > > > > allocated >> >> > > > > > > > by >> >> > > > > > > > > > > direct >> >> > > > > > > > > > > > > > buffers and we have nothing left for the JVM >> >> > > overhead. >> >> > > > So >> >> > > > > > we >> >> > > > > > > > > either >> >> > > > > > > > > > > > need >> >> > > > > > > > > > > > > a >> >> > > > > > > > > > > > > > way to compensate for that (again some safety >> >> > margin >> >> > > > > cutoff >> >> > > > > > > > > value) >> >> > > > > > > > > > or >> >> > > > > > > > > > > > we >> >> > > > > > > > > > > > > > will exceed container memory. >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > If we go with option 1.2, we need to be aware >> >> that >> >> > it >> >> > > > > takes >> >> > > > > > > > > > elaborate >> >> > > > > > > > > > > > > logic >> >> > > > > > > > > > > > > > to push recycling of direct buffers without >> >> always >> >> > > > > > > triggering a >> >> > > > > > > > > > full >> >> > > > > > > > > > > > GC. >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > My first guess is that the options will be >> >> easiest >> >> > to >> >> > > > do >> >> > > > > in >> >> > > > > > > the >> >> > > > > > > > > > > > following >> >> > > > > > > > > > > > > > order: >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 1.1 with a dedicated direct_memory >> >> > > > parameter, >> >> > > > > as >> >> > > > > > > > > > discussed >> >> > > > > > > > > > > > > > above. We would need to find a way to set the >> >> > > > > direct_memory >> >> > > > > > > > > > parameter >> >> > > > > > > > > > > > by >> >> > > > > > > > > > > > > > default. We could start with 64 MB and see >> how >> >> it >> >> > > goes >> >> > > > in >> >> > > > > > > > > practice. >> >> > > > > > > > > > > One >> >> > > > > > > > > > > > > > danger I see is that setting this loo low can >> >> > cause a >> >> > > > > bunch >> >> > > > > > > of >> >> > > > > > > > > > > > additional >> >> > > > > > > > > > > > > > GCs compared to before (we need to watch this >> >> > > > carefully). >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 2. It is actually quite simple to >> >> > > implement, >> >> > > > > we >> >> > > > > > > > could >> >> > > > > > > > > > try >> >> > > > > > > > > > > > how >> >> > > > > > > > > > > > > > segfault safe we are at the moment. >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > - Option 1.2: We would not touch the >> >> > > > > > > > "-XX:MaxDirectMemorySize" >> >> > > > > > > > > > > > > parameter >> >> > > > > > > > > > > > > > at all and assume that all the direct memory >> >> > > > allocations >> >> > > > > > that >> >> > > > > > > > the >> >> > > > > > > > > > JVM >> >> > > > > > > > > > > > and >> >> > > > > > > > > > > > > > Netty do are infrequent enough to be cleaned >> up >> >> > fast >> >> > > > > enough >> >> > > > > > > > > through >> >> > > > > > > > > > > > > regular >> >> > > > > > > > > > > > > > GC. I am not sure if that is a valid >> assumption, >> >> > > > though. >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > Best, >> >> > > > > > > > > > > > > > Stephan >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song >> < >> >> > > > > > > > > > tonysong...@gmail.com> >> >> > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thanks for sharing your opinion Till. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was >> >> > wondering >> >> > > > > > whether >> >> > > > > > > > we >> >> > > > > > > > > > can >> >> > > > > > > > > > > > > avoid >> >> > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap >> managed >> >> > memory >> >> > > > and >> >> > > > > > > > network >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > with >> >> > > > > > > > > > > > > > > alternative 3. But after giving it a second >> >> > > thought, >> >> > > > I >> >> > > > > > > think >> >> > > > > > > > > even >> >> > > > > > > > > > > for >> >> > > > > > > > > > > > > > > alternative 3 using direct memory for >> off-heap >> >> > > > managed >> >> > > > > > > memory >> >> > > > > > > > > > could >> >> > > > > > > > > > > > > cause >> >> > > > > > > > > > > > > > > problems. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Hi Yang, >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Regarding your concern, I think what >> proposed >> >> in >> >> > > this >> >> > > > > > FLIP >> >> > > > > > > it >> >> > > > > > > > > to >> >> > > > > > > > > > > have >> >> > > > > > > > > > > > > > both >> >> > > > > > > > > > > > > > > off-heap managed memory and network memory >> >> > > allocated >> >> > > > > > > through >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which means they are >> >> > practically >> >> > > > > > native >> >> > > > > > > > > memory >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > limited by JVM max direct memory. The only >> >> parts >> >> > of >> >> > > > > > memory >> >> > > > > > > > > > limited >> >> > > > > > > > > > > by >> >> > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > max direct memory are task off-heap memory >> and >> >> > JVM >> >> > > > > > > overhead, >> >> > > > > > > > > > which >> >> > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > exactly alternative 2 suggests to set the >> JVM >> >> max >> >> > > > > direct >> >> > > > > > > > memory >> >> > > > > > > > > > to. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till >> Rohrmann >> >> < >> >> > > > > > > > > > > trohrm...@apache.org> >> >> > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I >> >> > > understand >> >> > > > > the >> >> > > > > > > two >> >> > > > > > > > > > > > > alternatives >> >> > > > > > > > > > > > > > > > now. >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > I would be in favour of option 2 because >> it >> >> > makes >> >> > > > > > things >> >> > > > > > > > > > > explicit. >> >> > > > > > > > > > > > If >> >> > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > don't limit the direct memory, I fear >> that >> >> we >> >> > > might >> >> > > > > end >> >> > > > > > > up >> >> > > > > > > > > in a >> >> > > > > > > > > > > > > similar >> >> > > > > > > > > > > > > > > > situation as we are currently in: The >> user >> >> > might >> >> > > > see >> >> > > > > > that >> >> > > > > > > > her >> >> > > > > > > > > > > > process >> >> > > > > > > > > > > > > > > gets >> >> > > > > > > > > > > > > > > > killed by the OS and does not know why >> this >> >> is >> >> > > the >> >> > > > > > case. >> >> > > > > > > > > > > > > Consequently, >> >> > > > > > > > > > > > > > > she >> >> > > > > > > > > > > > > > > > tries to decrease the process memory size >> >> > > (similar >> >> > > > to >> >> > > > > > > > > > increasing >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > cutoff >> >> > > > > > > > > > > > > > > > ratio) in order to accommodate for the >> extra >> >> > > direct >> >> > > > > > > memory. >> >> > > > > > > > > > Even >> >> > > > > > > > > > > > > worse, >> >> > > > > > > > > > > > > > > she >> >> > > > > > > > > > > > > > > > tries to decrease memory budgets which >> are >> >> not >> >> > > > fully >> >> > > > > > used >> >> > > > > > > > and >> >> > > > > > > > > > > hence >> >> > > > > > > > > > > > > > won't >> >> > > > > > > > > > > > > > > > change the overall memory consumption. >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong >> >> Song < >> >> > > > > > > > > > > > tonysong...@gmail.com >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let me explain this with a concrete >> >> example >> >> > > Till. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let's say we have the following >> scenario. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Total Process Memory: 1GB >> >> > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap >> Memory + >> >> JVM >> >> > > > > > > Overhead): >> >> > > > > > > > > > 200MB >> >> > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM >> >> Metaspace, >> >> > > > > > Off-Heap >> >> > > > > > > > > > Managed >> >> > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > Network Memory): 800MB >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > For alternative 2, we set >> >> > > -XX:MaxDirectMemorySize >> >> > > > > to >> >> > > > > > > > 200MB. >> >> > > > > > > > > > > > > > > > > For alternative 3, we set >> >> > > -XX:MaxDirectMemorySize >> >> > > > > to >> >> > > > > > a >> >> > > > > > > > very >> >> > > > > > > > > > > large >> >> > > > > > > > > > > > > > > value, >> >> > > > > > > > > > > > > > > > > let's say 1TB. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of >> Task >> >> > > > Off-Heap >> >> > > > > > > Memory >> >> > > > > > > > > and >> >> > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > Overhead >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, then alternative 2 >> >> and >> >> > > > > > > alternative 3 >> >> > > > > > > > > > > should >> >> > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > same utility. Setting larger >> >> > > > > -XX:MaxDirectMemorySize >> >> > > > > > > will >> >> > > > > > > > > not >> >> > > > > > > > > > > > > reduce >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > sizes of the other memory pools. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of >> Task >> >> > > > Off-Heap >> >> > > > > > > Memory >> >> > > > > > > > > and >> >> > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, then >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > - Alternative 2 suffers from >> frequent >> >> OOM. >> >> > > To >> >> > > > > > avoid >> >> > > > > > > > > that, >> >> > > > > > > > > > > the >> >> > > > > > > > > > > > > only >> >> > > > > > > > > > > > > > > > thing >> >> > > > > > > > > > > > > > > > > user can do is to modify the >> >> configuration >> >> > > and >> >> > > > > > > > increase >> >> > > > > > > > > > JVM >> >> > > > > > > > > > > > > Direct >> >> > > > > > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM >> Overhead). >> >> > Let's >> >> > > > say >> >> > > > > > > that >> >> > > > > > > > > user >> >> > > > > > > > > > > > > > increases >> >> > > > > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will >> >> reduce >> >> > the >> >> > > > > total >> >> > > > > > > > size >> >> > > > > > > > > of >> >> > > > > > > > > > > > other >> >> > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > pools to 750MB, given the total >> process >> >> > > memory >> >> > > > > > > remains >> >> > > > > > > > > > 1GB. >> >> > > > > > > > > > > > > > > > > - For alternative 3, there is no >> >> chance of >> >> > > > > direct >> >> > > > > > > OOM. >> >> > > > > > > > > > There >> >> > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > chances >> >> > > > > > > > > > > > > > > > > of exceeding the total process >> memory >> >> > limit, >> >> > > > but >> >> > > > > > > given >> >> > > > > > > > > > that >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > process >> >> > > > > > > > > > > > > > > > > may >> >> > > > > > > > > > > > > > > > > not use up all the reserved native >> >> memory >> >> > > > > > (Off-Heap >> >> > > > > > > > > > Managed >> >> > > > > > > > > > > > > > Memory, >> >> > > > > > > > > > > > > > > > > Network >> >> > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the >> actual >> >> > direct >> >> > > > > > memory >> >> > > > > > > > > usage >> >> > > > > > > > > > is >> >> > > > > > > > > > > > > > > slightly >> >> > > > > > > > > > > > > > > > > above >> >> > > > > > > > > > > > > > > > > yet very close to 200MB, user >> probably >> >> do >> >> > > not >> >> > > > > need >> >> > > > > > > to >> >> > > > > > > > > > change >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > configurations. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Therefore, I think from the user's >> >> > > perspective, a >> >> > > > > > > > feasible >> >> > > > > > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > > > > for alternative 2 may lead to lower >> >> resource >> >> > > > > > > utilization >> >> > > > > > > > > > > compared >> >> > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > alternative 3. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till >> >> > Rohrmann >> >> > > < >> >> > > > > > > > > > > > > trohrm...@apache.org >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > I guess you have to help me >> understand >> >> the >> >> > > > > > difference >> >> > > > > > > > > > between >> >> > > > > > > > > > > > > > > > > alternative 2 >> >> > > > > > > > > > > > > > > > > > and 3 wrt to memory under utilization >> >> > > Xintong. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: set >> >> XX:MaxDirectMemorySize >> >> > > to >> >> > > > > Task >> >> > > > > > > > > > Off-Heap >> >> > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > > Overhead. Then there is the risk that >> >> this >> >> > > size >> >> > > > > is >> >> > > > > > > too >> >> > > > > > > > > low >> >> > > > > > > > > > > > > > resulting >> >> > > > > > > > > > > > > > > > in a >> >> > > > > > > > > > > > > > > > > > lot of garbage collection and >> >> potentially >> >> > an >> >> > > > OOM. >> >> > > > > > > > > > > > > > > > > > - Alternative 3: set >> >> XX:MaxDirectMemorySize >> >> > > to >> >> > > > > > > > something >> >> > > > > > > > > > > larger >> >> > > > > > > > > > > > > > than >> >> > > > > > > > > > > > > > > > > > alternative 2. This would of course >> >> reduce >> >> > > the >> >> > > > > > sizes >> >> > > > > > > of >> >> > > > > > > > > the >> >> > > > > > > > > > > > other >> >> > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > types. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > How would alternative 2 now result >> in an >> >> > > under >> >> > > > > > > > > utilization >> >> > > > > > > > > > of >> >> > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > compared to alternative 3? If >> >> alternative 3 >> >> > > > > > strictly >> >> > > > > > > > > sets a >> >> > > > > > > > > > > > > higher >> >> > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > direct memory size and we use only >> >> little, >> >> > > > then I >> >> > > > > > > would >> >> > > > > > > > > > > expect >> >> > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > alternative 3 results in memory under >> >> > > > > utilization. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang >> >> Wang < >> >> > > > > > > > > > > > danrtsey...@gmail.com >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Native and Direct Memory >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > My point is setting a very large >> max >> >> > direct >> >> > > > > > memory >> >> > > > > > > > size >> >> > > > > > > > > > > when >> >> > > > > > > > > > > > we >> >> > > > > > > > > > > > > > do >> >> > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > differentiate direct and native >> >> memory. >> >> > If >> >> > > > the >> >> > > > > > > direct >> >> > > > > > > > > > > > > > > > memory,including >> >> > > > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > direct memory and framework direct >> >> > > > memory,could >> >> > > > > > be >> >> > > > > > > > > > > calculated >> >> > > > > > > > > > > > > > > > > > > correctly,then >> >> > > > > > > > > > > > > > > > > > > i am in favor of setting direct >> memory >> >> > with >> >> > > > > fixed >> >> > > > > > > > > value. >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Memory Calculation >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn and >> >> k8s,we >> >> > > > need >> >> > > > > to >> >> > > > > > > > check >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > configurations in client to avoid >> >> > > submitting >> >> > > > > > > > > successfully >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > > > failing >> >> > > > > > > > > > > > > > > > > in >> >> > > > > > > > > > > > > > > > > > > the flink master. >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Best, >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Yang >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < >> tonysong...@gmail.com >> >> > > > > >于2019年8月13日 >> >> > > > > > > > > > 周二22:07写道： >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About MemorySegment, I think you >> are >> >> > > right >> >> > > > > that >> >> > > > > > > we >> >> > > > > > > > > > should >> >> > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > include >> >> > > > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > issue in the scope of this FLIP. >> >> This >> >> > > FLIP >> >> > > > > > should >> >> > > > > > > > > > > > concentrate >> >> > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > configure memory pools for >> >> > TaskExecutors, >> >> > > > > with >> >> > > > > > > > > minimum >> >> > > > > > > > > > > > > > > involvement >> >> > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > > > > > memory consumers use it. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About direct memory, I think >> >> > alternative >> >> > > 3 >> >> > > > > may >> >> > > > > > > not >> >> > > > > > > > > > having >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > same >> >> > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > reservation issue that >> alternative 2 >> >> > > does, >> >> > > > > but >> >> > > > > > at >> >> > > > > > > > the >> >> > > > > > > > > > > cost >> >> > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > risk >> >> > > > > > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > using memory at the container >> level, >> >> > > which >> >> > > > is >> >> > > > > > not >> >> > > > > > > > > good. >> >> > > > > > > > > > > My >> >> > > > > > > > > > > > > > point >> >> > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and >> "JVM >> >> > > > > Overhead" >> >> > > > > > > are >> >> > > > > > > > > not >> >> > > > > > > > > > > easy >> >> > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > config. >> >> > > > > > > > > > > > > > > > > > > For >> >> > > > > > > > > > > > > > > > > > > > alternative 2, users might >> configure >> >> > them >> >> > > > > > higher >> >> > > > > > > > than >> >> > > > > > > > > > > what >> >> > > > > > > > > > > > > > > actually >> >> > > > > > > > > > > > > > > > > > > needed, >> >> > > > > > > > > > > > > > > > > > > > just to avoid getting a direct >> OOM. >> >> For >> >> > > > > > > alternative >> >> > > > > > > > > 3, >> >> > > > > > > > > > > > users >> >> > > > > > > > > > > > > do >> >> > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > get >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so they may not >> config >> >> the >> >> > > two >> >> > > > > > > options >> >> > > > > > > > > > > > > aggressively >> >> > > > > > > > > > > > > > > > high. >> >> > > > > > > > > > > > > > > > > > But >> >> > > > > > > > > > > > > > > > > > > > the consequences are risks of >> >> overall >> >> > > > > container >> >> > > > > > > > > memory >> >> > > > > > > > > > > > usage >> >> > > > > > > > > > > > > > > > exceeds >> >> > > > > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > > > > budget. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM >> Till >> >> > > > > Rohrmann < >> >> > > > > > > > > > > > > > > > trohrm...@apache.org> >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for proposing this FLIP >> >> > Xintong. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > All in all I think it already >> >> looks >> >> > > quite >> >> > > > > > good. >> >> > > > > > > > > > > > Concerning >> >> > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > first >> >> > > > > > > > > > > > > > > > > > > open >> >> > > > > > > > > > > > > > > > > > > > > question about allocating >> memory >> >> > > > segments, >> >> > > > > I >> >> > > > > > > was >> >> > > > > > > > > > > > wondering >> >> > > > > > > > > > > > > > > > whether >> >> > > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > strictly necessary to do in the >> >> > context >> >> > > > of >> >> > > > > > this >> >> > > > > > > > > FLIP >> >> > > > > > > > > > or >> >> > > > > > > > > > > > > > whether >> >> > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > > > > > > > be done as a follow up? Without >> >> > knowing >> >> > > > all >> >> > > > > > > > > details, >> >> > > > > > > > > > I >> >> > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > > concerned >> >> > > > > > > > > > > > > > > > > > > > > that we would widen the scope >> of >> >> this >> >> > > > FLIP >> >> > > > > > too >> >> > > > > > > > much >> >> > > > > > > > > > > > because >> >> > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > > > > > > > to touch all the existing call >> >> sites >> >> > of >> >> > > > the >> >> > > > > > > > > > > MemoryManager >> >> > > > > > > > > > > > > > where >> >> > > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > > > allocate >> >> > > > > > > > > > > > > > > > > > > > > memory segments (this should >> >> mainly >> >> > be >> >> > > > > batch >> >> > > > > > > > > > > operators). >> >> > > > > > > > > > > > > The >> >> > > > > > > > > > > > > > > > > addition >> >> > > > > > > > > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > > > > > > > the memory reservation call to >> the >> >> > > > > > > MemoryManager >> >> > > > > > > > > > should >> >> > > > > > > > > > > > not >> >> > > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > affected >> >> > > > > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > > this and I would hope that >> this is >> >> > the >> >> > > > only >> >> > > > > > > point >> >> > > > > > > > > of >> >> > > > > > > > > > > > > > > interaction >> >> > > > > > > > > > > > > > > > a >> >> > > > > > > > > > > > > > > > > > > > > streaming job would have with >> the >> >> > > > > > > MemoryManager. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Concerning the second open >> >> question >> >> > > about >> >> > > > > > > setting >> >> > > > > > > > > or >> >> > > > > > > > > > > not >> >> > > > > > > > > > > > > > > setting >> >> > > > > > > > > > > > > > > > a >> >> > > > > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > > > > direct memory limit, I would >> also >> >> be >> >> > > > > > interested >> >> > > > > > > > why >> >> > > > > > > > > > > Yang >> >> > > > > > > > > > > > > Wang >> >> > > > > > > > > > > > > > > > > thinks >> >> > > > > > > > > > > > > > > > > > > > > leaving it open would be best. >> My >> >> > > concern >> >> > > > > > about >> >> > > > > > > > > this >> >> > > > > > > > > > > > would >> >> > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > > > > > > > > be in a similar situation as we >> >> are >> >> > now >> >> > > > > with >> >> > > > > > > the >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. >> >> > > > > > > > > > > > > > > > > > > If >> >> > > > > > > > > > > > > > > > > > > > > the different memory pools are >> not >> >> > > > clearly >> >> > > > > > > > > separated >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > can >> >> > > > > > > > > > > > > > > > spill >> >> > > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > > a different pool, then it is >> quite >> >> > hard >> >> > > > to >> >> > > > > > > > > understand >> >> > > > > > > > > > > > what >> >> > > > > > > > > > > > > > > > exactly >> >> > > > > > > > > > > > > > > > > > > > causes a >> >> > > > > > > > > > > > > > > > > > > > > process to get killed for using >> >> too >> >> > > much >> >> > > > > > > memory. >> >> > > > > > > > > This >> >> > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > then >> >> > > > > > > > > > > > > > > > > > easily >> >> > > > > > > > > > > > > > > > > > > > > lead to a similar situation >> what >> >> we >> >> > > have >> >> > > > > with >> >> > > > > > > the >> >> > > > > > > > > > > > > > cutoff-ratio. >> >> > > > > > > > > > > > > > > > So >> >> > > > > > > > > > > > > > > > > > why >> >> > > > > > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > > > setting a sane default value >> for >> >> max >> >> > > > direct >> >> > > > > > > > memory >> >> > > > > > > > > > and >> >> > > > > > > > > > > > > giving >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > an >> >> > > > > > > > > > > > > > > > > > > > > option to increase it if he >> runs >> >> into >> >> > > an >> >> > > > > OOM. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, how would >> alternative 2 >> >> > lead >> >> > > to >> >> > > > > > lower >> >> > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > utilization >> >> > > > > > > > > > > > > > > > > > than >> >> > > > > > > > > > > > > > > > > > > > > alternative 3 where we set the >> >> direct >> >> > > > > memory >> >> > > > > > > to a >> >> > > > > > > > > > > higher >> >> > > > > > > > > > > > > > value? >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM >> >> > Xintong >> >> > > > > Song < >> >> > > > > > > > > > > > > > > > tonysong...@gmail.com >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, >> Yang. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding your comments: >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* >> >> > > > > > > > > > > > > > > > > > > > > > I think setting a very large >> max >> >> > > direct >> >> > > > > > > memory >> >> > > > > > > > > size >> >> > > > > > > > > > > > > > > definitely >> >> > > > > > > > > > > > > > > > > has >> >> > > > > > > > > > > > > > > > > > > some >> >> > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not >> >> worry >> >> > > about >> >> > > > > > > direct >> >> > > > > > > > > OOM, >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > don't >> >> > > > > > > > > > > > > > > > > > even >> >> > > > > > > > > > > > > > > > > > > > > need >> >> > > > > > > > > > > > > > > > > > > > > > to allocate managed / network >> >> > memory >> >> > > > with >> >> > > > > > > > > > > > > > Unsafe.allocate() . >> >> > > > > > > > > > > > > > > > > > > > > > However, there are also some >> >> down >> >> > > sides >> >> > > > > of >> >> > > > > > > > doing >> >> > > > > > > > > > > this. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > - One thing I can think >> of is >> >> > that >> >> > > > if >> >> > > > > a >> >> > > > > > > task >> >> > > > > > > > > > > > executor >> >> > > > > > > > > > > > > > > > > container >> >> > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > > killed due to overusing >> >> memory, >> >> > it >> >> > > > > could >> >> > > > > > > be >> >> > > > > > > > > hard >> >> > > > > > > > > > > for >> >> > > > > > > > > > > > > use >> >> > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > know >> >> > > > > > > > > > > > > > > > > > > > which >> >> > > > > > > > > > > > > > > > > > > > > > part >> >> > > > > > > > > > > > > > > > > > > > > > of the memory is overused. >> >> > > > > > > > > > > > > > > > > > > > > > - Another down side is >> that >> >> the >> >> > > JVM >> >> > > > > > never >> >> > > > > > > > > > trigger >> >> > > > > > > > > > > GC >> >> > > > > > > > > > > > > due >> >> > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > reaching >> >> > > > > > > > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > > > > > direct memory limit, >> because >> >> the >> >> > > > limit >> >> > > > > > is >> >> > > > > > > > too >> >> > > > > > > > > > high >> >> > > > > > > > > > > > to >> >> > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > reached. >> >> > > > > > > > > > > > > > > > > > > > That >> >> > > > > > > > > > > > > > > > > > > > > > means we kind of relay on >> >> heap >> >> > > > memory >> >> > > > > to >> >> > > > > > > > > trigger >> >> > > > > > > > > > > GC >> >> > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > release >> >> > > > > > > > > > > > > > > > > > > > direct >> >> > > > > > > > > > > > > > > > > > > > > > memory. That could be a >> >> problem >> >> > in >> >> > > > > cases >> >> > > > > > > > where >> >> > > > > > > > > > we >> >> > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > more >> >> > > > > > > > > > > > > > > > > > direct >> >> > > > > > > > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > > > > usage but not enough heap >> >> > activity >> >> > > > to >> >> > > > > > > > trigger >> >> > > > > > > > > > the >> >> > > > > > > > > > > > GC. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you can share your >> reasons >> >> > for >> >> > > > > > > preferring >> >> > > > > > > > > > > > setting a >> >> > > > > > > > > > > > > > > very >> >> > > > > > > > > > > > > > > > > > large >> >> > > > > > > > > > > > > > > > > > > > > value, >> >> > > > > > > > > > > > > > > > > > > > > > if there are anything else I >> >> > > > overlooked. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* >> >> > > > > > > > > > > > > > > > > > > > > > If there is any conflict >> between >> >> > > > multiple >> >> > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > > > > explicitly specified, I >> think we >> >> > > should >> >> > > > > > throw >> >> > > > > > > > an >> >> > > > > > > > > > > error. >> >> > > > > > > > > > > > > > > > > > > > > > I think doing checking on the >> >> > client >> >> > > > side >> >> > > > > > is >> >> > > > > > > a >> >> > > > > > > > > good >> >> > > > > > > > > > > > idea, >> >> > > > > > > > > > > > > > so >> >> > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > > > > Yarn / >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can discover the >> problem >> >> > > before >> >> > > > > > > > submitting >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > Flink >> >> > > > > > > > > > > > > > > > > > cluster, >> >> > > > > > > > > > > > > > > > > > > > > which >> >> > > > > > > > > > > > > > > > > > > > > > is always a good thing. >> >> > > > > > > > > > > > > > > > > > > > > > But we can not only rely on >> the >> >> > > client >> >> > > > > side >> >> > > > > > > > > > checking, >> >> > > > > > > > > > > > > > because >> >> > > > > > > > > > > > > > > > for >> >> > > > > > > > > > > > > > > > > > > > > > standalone cluster >> TaskManagers >> >> on >> >> > > > > > different >> >> > > > > > > > > > machines >> >> > > > > > > > > > > > may >> >> > > > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > > > > > > different >> >> > > > > > > > > > > > > > > > > > > > > > configurations and the client >> >> does >> >> > > see >> >> > > > > > that. >> >> > > > > > > > > > > > > > > > > > > > > > What do you think? >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 >> PM >> >> Yang >> >> > > > Wang >> >> > > > > < >> >> > > > > > > > > > > > > > > > danrtsey...@gmail.com> >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi xintong, >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed >> >> > proposal. >> >> > > > > After >> >> > > > > > > all >> >> > > > > > > > > the >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > > > > > > > > introduced, it will be more >> >> > > powerful >> >> > > > to >> >> > > > > > > > control >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > flink >> >> > > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > > > usage. I >> >> > > > > > > > > > > > > > > > > > > > > > > just have few questions >> about >> >> it. >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Native and Direct >> Memory >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > We do not differentiate >> user >> >> > direct >> >> > > > > > memory >> >> > > > > > > > and >> >> > > > > > > > > > > native >> >> > > > > > > > > > > > > > > memory. >> >> > > > > > > > > > > > > > > > > > They >> >> > > > > > > > > > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > > > > > > > all >> >> > > > > > > > > > > > > > > > > > > > > > > included in task off-heap >> >> memory. >> >> > > > > Right? >> >> > > > > > > So i >> >> > > > > > > > > > don’t >> >> > > > > > > > > > > > > think >> >> > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > > > > set >> >> > > > > > > > > > > > > > > > > > > > > > > the -XX:MaxDirectMemorySize >> >> > > > properly. I >> >> > > > > > > > prefer >> >> > > > > > > > > > > > leaving >> >> > > > > > > > > > > > > > it a >> >> > > > > > > > > > > > > > > > > very >> >> > > > > > > > > > > > > > > > > > > > large >> >> > > > > > > > > > > > > > > > > > > > > > > value. >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > If the sum of and >> fine-grained >> >> > > > > > > memory(network >> >> > > > > > > > > > > memory, >> >> > > > > > > > > > > > > > > managed >> >> > > > > > > > > > > > > > > > > > > memory, >> >> > > > > > > > > > > > > > > > > > > > > > etc.) >> >> > > > > > > > > > > > > > > > > > > > > > > is larger than total >> process >> >> > > memory, >> >> > > > > how >> >> > > > > > do >> >> > > > > > > > we >> >> > > > > > > > > > deal >> >> > > > > > > > > > > > > with >> >> > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > > situation? >> >> > > > > > > > > > > > > > > > > > > > > > Do >> >> > > > > > > > > > > > > > > > > > > > > > > we need to check the memory >> >> > > > > configuration >> >> > > > > > > in >> >> > > > > > > > > > > client? >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong Song < >> >> > > tonysong...@gmail.com> >> >> > > > > > > > > > 于2019年8月7日周三 >> >> > > > > > > > > > > > > > > 下午10:14写道： >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > We would like to start a >> >> > > discussion >> >> > > > > > > thread >> >> > > > > > > > on >> >> > > > > > > > > > > > > "FLIP-49: >> >> > > > > > > > > > > > > > > > > Unified >> >> > > > > > > > > > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > > > > > > > > > > Configuration for >> >> > > > TaskExecutors"[1], >> >> > > > > > > where >> >> > > > > > > > we >> >> > > > > > > > > > > > > describe >> >> > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > improve >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory >> >> > > configurations. >> >> > > > > The >> >> > > > > > > > FLIP >> >> > > > > > > > > > > > document >> >> > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > mostly >> >> > > > > > > > > > > > > > > > > > > > based >> >> > > > > > > > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > > > > > > an >> >> > > > > > > > > > > > > > > > > > > > > > > > early design "Memory >> >> Management >> >> > > and >> >> > > > > > > > > > Configuration >> >> > > > > > > > > > > > > > > > > Reloaded"[2] >> >> > > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, >> >> > > > > > > > > > > > > > > > > > > > > > > > with updates from >> follow-up >> >> > > > > discussions >> >> > > > > > > > both >> >> > > > > > > > > > > online >> >> > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > > offline. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses >> several >> >> > > > > > shortcomings >> >> > > > > > > of >> >> > > > > > > > > > > current >> >> > > > > > > > > > > > > > > (Flink >> >> > > > > > > > > > > > > > > > > 1.9) >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory >> >> > > configuration. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Different >> configuration >> >> > for >> >> > > > > > > Streaming >> >> > > > > > > > > and >> >> > > > > > > > > > > > Batch. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complex and >> difficult >> >> > > > > > configuration >> >> > > > > > > of >> >> > > > > > > > > > > RocksDB >> >> > > > > > > > > > > > > in >> >> > > > > > > > > > > > > > > > > > Streaming. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complicated, >> uncertain >> >> and >> >> > > > hard >> >> > > > > to >> >> > > > > > > > > > > understand. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve the >> >> > problems >> >> > > > can >> >> > > > > > be >> >> > > > > > > > > > > summarized >> >> > > > > > > > > > > > > as >> >> > > > > > > > > > > > > > > > > follows. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Extend memory >> manager >> >> to >> >> > > also >> >> > > > > > > account >> >> > > > > > > > > for >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > usage >> >> > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > state >> >> > > > > > > > > > > > > > > > > > > > > > > > backends. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Modify how >> TaskExecutor >> >> > > memory >> >> > > > > is >> >> > > > > > > > > > > partitioned >> >> > > > > > > > > > > > > > > > accounted >> >> > > > > > > > > > > > > > > > > > > > > individual >> >> > > > > > > > > > > > > > > > > > > > > > > > memory reservations >> and >> >> > pools. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory >> >> > > configuration >> >> > > > > > > options >> >> > > > > > > > > and >> >> > > > > > > > > > > > > > > calculations >> >> > > > > > > > > > > > > > > > > > > logics. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please find more details >> in >> >> the >> >> > > > FLIP >> >> > > > > > wiki >> >> > > > > > > > > > > document >> >> > > > > > > > > > > > > [1]. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > (Please note that the >> early >> >> > > design >> >> > > > > doc >> >> > > > > > > [2] >> >> > > > > > > > is >> >> > > > > > > > > > out >> >> > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > sync, >> >> > > > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > > it >> >> > > > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the >> >> > > discussion >> >> > > > in >> >> > > > > > > this >> >> > > > > > > > > > > mailing >> >> > > > > > > > > > > > > list >> >> > > > > > > > > > > > > > > > > > thread.) >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your >> >> > > feedbacks. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song >> < >> >> > > > > > > > > > tonysong...@gmail.com> >> >> > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thanks for sharing your opinion Till. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was >> >> > wondering >> >> > > > > > whether >> >> > > > > > > > we >> >> > > > > > > > > > can >> >> > > > > > > > > > > > > avoid >> >> > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap >> managed >> >> > memory >> >> > > > and >> >> > > > > > > > network >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > with >> >> > > > > > > > > > > > > > > alternative 3. But after giving it a second >> >> > > thought, >> >> > > > I >> >> > > > > > > think >> >> > > > > > > > > even >> >> > > > > > > > > > > for >> >> > > > > > > > > > > > > > > alternative 3 using direct memory for >> off-heap >> >> > > > managed >> >> > > > > > > memory >> >> > > > > > > > > > could >> >> > > > > > > > > > > > > cause >> >> > > > > > > > > > > > > > > problems. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Hi Yang, >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Regarding your concern, I think what >> proposed >> >> in >> >> > > this >> >> > > > > > FLIP >> >> > > > > > > it >> >> > > > > > > > > to >> >> > > > > > > > > > > have >> >> > > > > > > > > > > > > > both >> >> > > > > > > > > > > > > > > off-heap managed memory and network memory >> >> > > allocated >> >> > > > > > > through >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which means they are >> >> > practically >> >> > > > > > native >> >> > > > > > > > > memory >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > limited by JVM max direct memory. The only >> >> parts >> >> > of >> >> > > > > > memory >> >> > > > > > > > > > limited >> >> > > > > > > > > > > by >> >> > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > max direct memory are task off-heap memory >> and >> >> > JVM >> >> > > > > > > overhead, >> >> > > > > > > > > > which >> >> > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > exactly alternative 2 suggests to set the >> JVM >> >> max >> >> > > > > direct >> >> > > > > > > > memory >> >> > > > > > > > > > to. >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till >> Rohrmann >> >> < >> >> > > > > > > > > > > trohrm...@apache.org> >> >> > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I >> >> > > understand >> >> > > > > the >> >> > > > > > > two >> >> > > > > > > > > > > > > alternatives >> >> > > > > > > > > > > > > > > > now. >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > I would be in favour of option 2 because >> it >> >> > makes >> >> > > > > > things >> >> > > > > > > > > > > explicit. >> >> > > > > > > > > > > > If >> >> > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > don't limit the direct memory, I fear >> that >> >> we >> >> > > might >> >> > > > > end >> >> > > > > > > up >> >> > > > > > > > > in a >> >> > > > > > > > > > > > > similar >> >> > > > > > > > > > > > > > > > situation as we are currently in: The >> user >> >> > might >> >> > > > see >> >> > > > > > that >> >> > > > > > > > her >> >> > > > > > > > > > > > process >> >> > > > > > > > > > > > > > > gets >> >> > > > > > > > > > > > > > > > killed by the OS and does not know why >> this >> >> is >> >> > > the >> >> > > > > > case. >> >> > > > > > > > > > > > > Consequently, >> >> > > > > > > > > > > > > > > she >> >> > > > > > > > > > > > > > > > tries to decrease the process memory size >> >> > > (similar >> >> > > > to >> >> > > > > > > > > > increasing >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > cutoff >> >> > > > > > > > > > > > > > > > ratio) in order to accommodate for the >> extra >> >> > > direct >> >> > > > > > > memory. >> >> > > > > > > > > > Even >> >> > > > > > > > > > > > > worse, >> >> > > > > > > > > > > > > > > she >> >> > > > > > > > > > > > > > > > tries to decrease memory budgets which >> are >> >> not >> >> > > > fully >> >> > > > > > used >> >> > > > > > > > and >> >> > > > > > > > > > > hence >> >> > > > > > > > > > > > > > won't >> >> > > > > > > > > > > > > > > > change the overall memory consumption. >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong >> >> Song < >> >> > > > > > > > > > > > tonysong...@gmail.com >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let me explain this with a concrete >> >> example >> >> > > Till. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Let's say we have the following >> scenario. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Total Process Memory: 1GB >> >> > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap >> Memory + >> >> JVM >> >> > > > > > > Overhead): >> >> > > > > > > > > > 200MB >> >> > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM >> >> Metaspace, >> >> > > > > > Off-Heap >> >> > > > > > > > > > Managed >> >> > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > Network Memory): 800MB >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > For alternative 2, we set >> >> > > -XX:MaxDirectMemorySize >> >> > > > > to >> >> > > > > > > > 200MB. >> >> > > > > > > > > > > > > > > > > For alternative 3, we set >> >> > > -XX:MaxDirectMemorySize >> >> > > > > to >> >> > > > > > a >> >> > > > > > > > very >> >> > > > > > > > > > > large >> >> > > > > > > > > > > > > > > value, >> >> > > > > > > > > > > > > > > > > let's say 1TB. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of >> Task >> >> > > > Off-Heap >> >> > > > > > > Memory >> >> > > > > > > > > and >> >> > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > Overhead >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, then alternative 2 >> >> and >> >> > > > > > > alternative 3 >> >> > > > > > > > > > > should >> >> > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > same utility. Setting larger >> >> > > > > -XX:MaxDirectMemorySize >> >> > > > > > > will >> >> > > > > > > > > not >> >> > > > > > > > > > > > > reduce >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > sizes of the other memory pools. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > If the actual direct memory usage of >> Task >> >> > > > Off-Heap >> >> > > > > > > Memory >> >> > > > > > > > > and >> >> > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, then >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > - Alternative 2 suffers from >> frequent >> >> OOM. >> >> > > To >> >> > > > > > avoid >> >> > > > > > > > > that, >> >> > > > > > > > > > > the >> >> > > > > > > > > > > > > only >> >> > > > > > > > > > > > > > > > thing >> >> > > > > > > > > > > > > > > > > user can do is to modify the >> >> configuration >> >> > > and >> >> > > > > > > > increase >> >> > > > > > > > > > JVM >> >> > > > > > > > > > > > > Direct >> >> > > > > > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM >> Overhead). >> >> > Let's >> >> > > > say >> >> > > > > > > that >> >> > > > > > > > > user >> >> > > > > > > > > > > > > > increases >> >> > > > > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will >> >> reduce >> >> > the >> >> > > > > total >> >> > > > > > > > size >> >> > > > > > > > > of >> >> > > > > > > > > > > > other >> >> > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > pools to 750MB, given the total >> process >> >> > > memory >> >> > > > > > > remains >> >> > > > > > > > > > 1GB. >> >> > > > > > > > > > > > > > > > > - For alternative 3, there is no >> >> chance of >> >> > > > > direct >> >> > > > > > > OOM. >> >> > > > > > > > > > There >> >> > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > chances >> >> > > > > > > > > > > > > > > > > of exceeding the total process >> memory >> >> > limit, >> >> > > > but >> >> > > > > > > given >> >> > > > > > > > > > that >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > process >> >> > > > > > > > > > > > > > > > > may >> >> > > > > > > > > > > > > > > > > not use up all the reserved native >> >> memory >> >> > > > > > (Off-Heap >> >> > > > > > > > > > Managed >> >> > > > > > > > > > > > > > Memory, >> >> > > > > > > > > > > > > > > > > Network >> >> > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the >> actual >> >> > direct >> >> > > > > > memory >> >> > > > > > > > > usage >> >> > > > > > > > > > is >> >> > > > > > > > > > > > > > > slightly >> >> > > > > > > > > > > > > > > > > above >> >> > > > > > > > > > > > > > > > > yet very close to 200MB, user >> probably >> >> do >> >> > > not >> >> > > > > need >> >> > > > > > > to >> >> > > > > > > > > > change >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > configurations. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Therefore, I think from the user's >> >> > > perspective, a >> >> > > > > > > > feasible >> >> > > > > > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > > > > for alternative 2 may lead to lower >> >> resource >> >> > > > > > > utilization >> >> > > > > > > > > > > compared >> >> > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > alternative 3. >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till >> >> > Rohrmann >> >> > > < >> >> > > > > > > > > > > > > trohrm...@apache.org >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > I guess you have to help me >> understand >> >> the >> >> > > > > > difference >> >> > > > > > > > > > between >> >> > > > > > > > > > > > > > > > > alternative 2 >> >> > > > > > > > > > > > > > > > > > and 3 wrt to memory under utilization >> >> > > Xintong. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: set >> >> XX:MaxDirectMemorySize >> >> > > to >> >> > > > > Task >> >> > > > > > > > > > Off-Heap >> >> > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > JVM >> >> > > > > > > > > > > > > > > > > > Overhead. Then there is the risk that >> >> this >> >> > > size >> >> > > > > is >> >> > > > > > > too >> >> > > > > > > > > low >> >> > > > > > > > > > > > > > resulting >> >> > > > > > > > > > > > > > > > in a >> >> > > > > > > > > > > > > > > > > > lot of garbage collection and >> >> potentially >> >> > an >> >> > > > OOM. >> >> > > > > > > > > > > > > > > > > > - Alternative 3: set >> >> XX:MaxDirectMemorySize >> >> > > to >> >> > > > > > > > something >> >> > > > > > > > > > > larger >> >> > > > > > > > > > > > > > than >> >> > > > > > > > > > > > > > > > > > alternative 2. This would of course >> >> reduce >> >> > > the >> >> > > > > > sizes >> >> > > > > > > of >> >> > > > > > > > > the >> >> > > > > > > > > > > > other >> >> > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > types. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > How would alternative 2 now result >> in an >> >> > > under >> >> > > > > > > > > utilization >> >> > > > > > > > > > of >> >> > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > compared to alternative 3? If >> >> alternative 3 >> >> > > > > > strictly >> >> > > > > > > > > sets a >> >> > > > > > > > > > > > > higher >> >> > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > direct memory size and we use only >> >> little, >> >> > > > then I >> >> > > > > > > would >> >> > > > > > > > > > > expect >> >> > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > alternative 3 results in memory under >> >> > > > > utilization. >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang >> >> Wang < >> >> > > > > > > > > > > > danrtsey...@gmail.com >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Native and Direct Memory >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > My point is setting a very large >> max >> >> > direct >> >> > > > > > memory >> >> > > > > > > > size >> >> > > > > > > > > > > when >> >> > > > > > > > > > > > we >> >> > > > > > > > > > > > > > do >> >> > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > differentiate direct and native >> >> memory. >> >> > If >> >> > > > the >> >> > > > > > > direct >> >> > > > > > > > > > > > > > > > memory,including >> >> > > > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > direct memory and framework direct >> >> > > > memory,could >> >> > > > > > be >> >> > > > > > > > > > > calculated >> >> > > > > > > > > > > > > > > > > > > correctly,then >> >> > > > > > > > > > > > > > > > > > > i am in favor of setting direct >> memory >> >> > with >> >> > > > > fixed >> >> > > > > > > > > value. >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Memory Calculation >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn and >> >> k8s,we >> >> > > > need >> >> > > > > to >> >> > > > > > > > check >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > configurations in client to avoid >> >> > > submitting >> >> > > > > > > > > successfully >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > > > failing >> >> > > > > > > > > > > > > > > > > in >> >> > > > > > > > > > > > > > > > > > > the flink master. >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Best, >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Yang >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < >> tonysong...@gmail.com >> >> > > > > >于2019年8月13日 >> >> > > > > > > > > > 周二22:07写道： >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About MemorySegment, I think you >> are >> >> > > right >> >> > > > > that >> >> > > > > > > we >> >> > > > > > > > > > should >> >> > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > include >> >> > > > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > issue in the scope of this FLIP. >> >> This >> >> > > FLIP >> >> > > > > > should >> >> > > > > > > > > > > > concentrate >> >> > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > configure memory pools for >> >> > TaskExecutors, >> >> > > > > with >> >> > > > > > > > > minimum >> >> > > > > > > > > > > > > > > involvement >> >> > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > > > > > memory consumers use it. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > About direct memory, I think >> >> > alternative >> >> > > 3 >> >> > > > > may >> >> > > > > > > not >> >> > > > > > > > > > having >> >> > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > same >> >> > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > reservation issue that >> alternative 2 >> >> > > does, >> >> > > > > but >> >> > > > > > at >> >> > > > > > > > the >> >> > > > > > > > > > > cost >> >> > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > risk >> >> > > > > > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > using memory at the container >> level, >> >> > > which >> >> > > > is >> >> > > > > > not >> >> > > > > > > > > good. >> >> > > > > > > > > > > My >> >> > > > > > > > > > > > > > point >> >> > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and >> "JVM >> >> > > > > Overhead" >> >> > > > > > > are >> >> > > > > > > > > not >> >> > > > > > > > > > > easy >> >> > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > config. >> >> > > > > > > > > > > > > > > > > > > For >> >> > > > > > > > > > > > > > > > > > > > alternative 2, users might >> configure >> >> > them >> >> > > > > > higher >> >> > > > > > > > than >> >> > > > > > > > > > > what >> >> > > > > > > > > > > > > > > actually >> >> > > > > > > > > > > > > > > > > > > needed, >> >> > > > > > > > > > > > > > > > > > > > just to avoid getting a direct >> OOM. >> >> For >> >> > > > > > > alternative >> >> > > > > > > > > 3, >> >> > > > > > > > > > > > users >> >> > > > > > > > > > > > > do >> >> > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > get >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so they may not >> config >> >> the >> >> > > two >> >> > > > > > > options >> >> > > > > > > > > > > > > aggressively >> >> > > > > > > > > > > > > > > > high. >> >> > > > > > > > > > > > > > > > > > But >> >> > > > > > > > > > > > > > > > > > > > the consequences are risks of >> >> overall >> >> > > > > container >> >> > > > > > > > > memory >> >> > > > > > > > > > > > usage >> >> > > > > > > > > > > > > > > > exceeds >> >> > > > > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > > > > budget. >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM >> Till >> >> > > > > Rohrmann < >> >> > > > > > > > > > > > > > > > trohrm...@apache.org> >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for proposing this FLIP >> >> > Xintong. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > All in all I think it already >> >> looks >> >> > > quite >> >> > > > > > good. >> >> > > > > > > > > > > > Concerning >> >> > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > first >> >> > > > > > > > > > > > > > > > > > > open >> >> > > > > > > > > > > > > > > > > > > > > question about allocating >> memory >> >> > > > segments, >> >> > > > > I >> >> > > > > > > was >> >> > > > > > > > > > > > wondering >> >> > > > > > > > > > > > > > > > whether >> >> > > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > strictly necessary to do in the >> >> > context >> >> > > > of >> >> > > > > > this >> >> > > > > > > > > FLIP >> >> > > > > > > > > > or >> >> > > > > > > > > > > > > > whether >> >> > > > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > > > > > > > be done as a follow up? Without >> >> > knowing >> >> > > > all >> >> > > > > > > > > details, >> >> > > > > > > > > > I >> >> > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > > concerned >> >> > > > > > > > > > > > > > > > > > > > > that we would widen the scope >> of >> >> this >> >> > > > FLIP >> >> > > > > > too >> >> > > > > > > > much >> >> > > > > > > > > > > > because >> >> > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > > > > > > > to touch all the existing call >> >> sites >> >> > of >> >> > > > the >> >> > > > > > > > > > > MemoryManager >> >> > > > > > > > > > > > > > where >> >> > > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > > > allocate >> >> > > > > > > > > > > > > > > > > > > > > memory segments (this should >> >> mainly >> >> > be >> >> > > > > batch >> >> > > > > > > > > > > operators). >> >> > > > > > > > > > > > > The >> >> > > > > > > > > > > > > > > > > addition >> >> > > > > > > > > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > > > > > > > the memory reservation call to >> the >> >> > > > > > > MemoryManager >> >> > > > > > > > > > should >> >> > > > > > > > > > > > not >> >> > > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > affected >> >> > > > > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > > this and I would hope that >> this is >> >> > the >> >> > > > only >> >> > > > > > > point >> >> > > > > > > > > of >> >> > > > > > > > > > > > > > > interaction >> >> > > > > > > > > > > > > > > > a >> >> > > > > > > > > > > > > > > > > > > > > streaming job would have with >> the >> >> > > > > > > MemoryManager. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Concerning the second open >> >> question >> >> > > about >> >> > > > > > > setting >> >> > > > > > > > > or >> >> > > > > > > > > > > not >> >> > > > > > > > > > > > > > > setting >> >> > > > > > > > > > > > > > > > a >> >> > > > > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > > > > direct memory limit, I would >> also >> >> be >> >> > > > > > interested >> >> > > > > > > > why >> >> > > > > > > > > > > Yang >> >> > > > > > > > > > > > > Wang >> >> > > > > > > > > > > > > > > > > thinks >> >> > > > > > > > > > > > > > > > > > > > > leaving it open would be best. >> My >> >> > > concern >> >> > > > > > about >> >> > > > > > > > > this >> >> > > > > > > > > > > > would >> >> > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > > > would >> >> > > > > > > > > > > > > > > > > > > > > be in a similar situation as we >> >> are >> >> > now >> >> > > > > with >> >> > > > > > > the >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. >> >> > > > > > > > > > > > > > > > > > > If >> >> > > > > > > > > > > > > > > > > > > > > the different memory pools are >> not >> >> > > > clearly >> >> > > > > > > > > separated >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > can >> >> > > > > > > > > > > > > > > > spill >> >> > > > > > > > > > > > > > > > > > over >> >> > > > > > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > > a different pool, then it is >> quite >> >> > hard >> >> > > > to >> >> > > > > > > > > understand >> >> > > > > > > > > > > > what >> >> > > > > > > > > > > > > > > > exactly >> >> > > > > > > > > > > > > > > > > > > > causes a >> >> > > > > > > > > > > > > > > > > > > > > process to get killed for using >> >> too >> >> > > much >> >> > > > > > > memory. >> >> > > > > > > > > This >> >> > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > then >> >> > > > > > > > > > > > > > > > > > easily >> >> > > > > > > > > > > > > > > > > > > > > lead to a similar situation >> what >> >> we >> >> > > have >> >> > > > > with >> >> > > > > > > the >> >> > > > > > > > > > > > > > cutoff-ratio. >> >> > > > > > > > > > > > > > > > So >> >> > > > > > > > > > > > > > > > > > why >> >> > > > > > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > > > setting a sane default value >> for >> >> max >> >> > > > direct >> >> > > > > > > > memory >> >> > > > > > > > > > and >> >> > > > > > > > > > > > > giving >> >> > > > > > > > > > > > > > > the >> >> > > > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > an >> >> > > > > > > > > > > > > > > > > > > > > option to increase it if he >> runs >> >> into >> >> > > an >> >> > > > > OOM. >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, how would >> alternative 2 >> >> > lead >> >> > > to >> >> > > > > > lower >> >> > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > utilization >> >> > > > > > > > > > > > > > > > > > than >> >> > > > > > > > > > > > > > > > > > > > > alternative 3 where we set the >> >> direct >> >> > > > > memory >> >> > > > > > > to a >> >> > > > > > > > > > > higher >> >> > > > > > > > > > > > > > value? >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, >> >> > > > > > > > > > > > > > > > > > > > > Till >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM >> >> > Xintong >> >> > > > > Song < >> >> > > > > > > > > > > > > > > > tonysong...@gmail.com >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, >> Yang. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding your comments: >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* >> >> > > > > > > > > > > > > > > > > > > > > > I think setting a very large >> max >> >> > > direct >> >> > > > > > > memory >> >> > > > > > > > > size >> >> > > > > > > > > > > > > > > definitely >> >> > > > > > > > > > > > > > > > > has >> >> > > > > > > > > > > > > > > > > > > some >> >> > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not >> >> worry >> >> > > about >> >> > > > > > > direct >> >> > > > > > > > > OOM, >> >> > > > > > > > > > > and >> >> > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > don't >> >> > > > > > > > > > > > > > > > > > even >> >> > > > > > > > > > > > > > > > > > > > > need >> >> > > > > > > > > > > > > > > > > > > > > > to allocate managed / network >> >> > memory >> >> > > > with >> >> > > > > > > > > > > > > > Unsafe.allocate() . >> >> > > > > > > > > > > > > > > > > > > > > > However, there are also some >> >> down >> >> > > sides >> >> > > > > of >> >> > > > > > > > doing >> >> > > > > > > > > > > this. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > - One thing I can think >> of is >> >> > that >> >> > > > if >> >> > > > > a >> >> > > > > > > task >> >> > > > > > > > > > > > executor >> >> > > > > > > > > > > > > > > > > container >> >> > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > > killed due to overusing >> >> memory, >> >> > it >> >> > > > > could >> >> > > > > > > be >> >> > > > > > > > > hard >> >> > > > > > > > > > > for >> >> > > > > > > > > > > > > use >> >> > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > know >> >> > > > > > > > > > > > > > > > > > > > which >> >> > > > > > > > > > > > > > > > > > > > > > part >> >> > > > > > > > > > > > > > > > > > > > > > of the memory is overused. >> >> > > > > > > > > > > > > > > > > > > > > > - Another down side is >> that >> >> the >> >> > > JVM >> >> > > > > > never >> >> > > > > > > > > > trigger >> >> > > > > > > > > > > GC >> >> > > > > > > > > > > > > due >> >> > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > reaching >> >> > > > > > > > > > > > > > > > > > > > > max >> >> > > > > > > > > > > > > > > > > > > > > > direct memory limit, >> because >> >> the >> >> > > > limit >> >> > > > > > is >> >> > > > > > > > too >> >> > > > > > > > > > high >> >> > > > > > > > > > > > to >> >> > > > > > > > > > > > > be >> >> > > > > > > > > > > > > > > > > > reached. >> >> > > > > > > > > > > > > > > > > > > > That >> >> > > > > > > > > > > > > > > > > > > > > > means we kind of relay on >> >> heap >> >> > > > memory >> >> > > > > to >> >> > > > > > > > > trigger >> >> > > > > > > > > > > GC >> >> > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > release >> >> > > > > > > > > > > > > > > > > > > > direct >> >> > > > > > > > > > > > > > > > > > > > > > memory. That could be a >> >> problem >> >> > in >> >> > > > > cases >> >> > > > > > > > where >> >> > > > > > > > > > we >> >> > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > more >> >> > > > > > > > > > > > > > > > > > direct >> >> > > > > > > > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > > > > usage but not enough heap >> >> > activity >> >> > > > to >> >> > > > > > > > trigger >> >> > > > > > > > > > the >> >> > > > > > > > > > > > GC. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you can share your >> reasons >> >> > for >> >> > > > > > > preferring >> >> > > > > > > > > > > > setting a >> >> > > > > > > > > > > > > > > very >> >> > > > > > > > > > > > > > > > > > large >> >> > > > > > > > > > > > > > > > > > > > > value, >> >> > > > > > > > > > > > > > > > > > > > > > if there are anything else I >> >> > > > overlooked. >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* >> >> > > > > > > > > > > > > > > > > > > > > > If there is any conflict >> between >> >> > > > multiple >> >> > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > user >> >> > > > > > > > > > > > > > > > > > > > > > explicitly specified, I >> think we >> >> > > should >> >> > > > > > throw >> >> > > > > > > > an >> >> > > > > > > > > > > error. >> >> > > > > > > > > > > > > > > > > > > > > > I think doing checking on the >> >> > client >> >> > > > side >> >> > > > > > is >> >> > > > > > > a >> >> > > > > > > > > good >> >> > > > > > > > > > > > idea, >> >> > > > > > > > > > > > > > so >> >> > > > > > > > > > > > > > > > that >> >> > > > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > > > > Yarn / >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can discover the >> problem >> >> > > before >> >> > > > > > > > submitting >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > Flink >> >> > > > > > > > > > > > > > > > > > cluster, >> >> > > > > > > > > > > > > > > > > > > > > which >> >> > > > > > > > > > > > > > > > > > > > > > is always a good thing. >> >> > > > > > > > > > > > > > > > > > > > > > But we can not only rely on >> the >> >> > > client >> >> > > > > side >> >> > > > > > > > > > checking, >> >> > > > > > > > > > > > > > because >> >> > > > > > > > > > > > > > > > for >> >> > > > > > > > > > > > > > > > > > > > > > standalone cluster >> TaskManagers >> >> on >> >> > > > > > different >> >> > > > > > > > > > machines >> >> > > > > > > > > > > > may >> >> > > > > > > > > > > > > > > have >> >> > > > > > > > > > > > > > > > > > > > different >> >> > > > > > > > > > > > > > > > > > > > > > configurations and the client >> >> does >> >> > > see >> >> > > > > > that. >> >> > > > > > > > > > > > > > > > > > > > > > What do you think? >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 >> PM >> >> Yang >> >> > > > Wang >> >> > > > > < >> >> > > > > > > > > > > > > > > > danrtsey...@gmail.com> >> >> > > > > > > > > > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi xintong, >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed >> >> > proposal. >> >> > > > > After >> >> > > > > > > all >> >> > > > > > > > > the >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > configuration >> >> > > > > > > > > > > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > > > > > > > > introduced, it will be more >> >> > > powerful >> >> > > > to >> >> > > > > > > > control >> >> > > > > > > > > > the >> >> > > > > > > > > > > > > flink >> >> > > > > > > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > > > > > > > usage. I >> >> > > > > > > > > > > > > > > > > > > > > > > just have few questions >> about >> >> it. >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Native and Direct >> Memory >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > We do not differentiate >> user >> >> > direct >> >> > > > > > memory >> >> > > > > > > > and >> >> > > > > > > > > > > native >> >> > > > > > > > > > > > > > > memory. >> >> > > > > > > > > > > > > > > > > > They >> >> > > > > > > > > > > > > > > > > > > > are >> >> > > > > > > > > > > > > > > > > > > > > > all >> >> > > > > > > > > > > > > > > > > > > > > > > included in task off-heap >> >> memory. >> >> > > > > Right? >> >> > > > > > > So i >> >> > > > > > > > > > don’t >> >> > > > > > > > > > > > > think >> >> > > > > > > > > > > > > > > we >> >> > > > > > > > > > > > > > > > > > could >> >> > > > > > > > > > > > > > > > > > > > not >> >> > > > > > > > > > > > > > > > > > > > > > set >> >> > > > > > > > > > > > > > > > > > > > > > > the -XX:MaxDirectMemorySize >> >> > > > properly. I >> >> > > > > > > > prefer >> >> > > > > > > > > > > > leaving >> >> > > > > > > > > > > > > > it a >> >> > > > > > > > > > > > > > > > > very >> >> > > > > > > > > > > > > > > > > > > > large >> >> > > > > > > > > > > > > > > > > > > > > > > value. >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > If the sum of and >> fine-grained >> >> > > > > > > memory(network >> >> > > > > > > > > > > memory, >> >> > > > > > > > > > > > > > > managed >> >> > > > > > > > > > > > > > > > > > > memory, >> >> > > > > > > > > > > > > > > > > > > > > > etc.) >> >> > > > > > > > > > > > > > > > > > > > > > > is larger than total >> process >> >> > > memory, >> >> > > > > how >> >> > > > > > do >> >> > > > > > > > we >> >> > > > > > > > > > deal >> >> > > > > > > > > > > > > with >> >> > > > > > > > > > > > > > > this >> >> > > > > > > > > > > > > > > > > > > > > situation? >> >> > > > > > > > > > > > > > > > > > > > > > Do >> >> > > > > > > > > > > > > > > > > > > > > > > we need to check the memory >> >> > > > > configuration >> >> > > > > > > in >> >> > > > > > > > > > > client? >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong Song < >> >> > > tonysong...@gmail.com> >> >> > > > > > > > > > 于2019年8月7日周三 >> >> > > > > > > > > > > > > > > 下午10:14写道： >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > We would like to start a >> >> > > discussion >> >> > > > > > > thread >> >> > > > > > > > on >> >> > > > > > > > > > > > > "FLIP-49: >> >> > > > > > > > > > > > > > > > > Unified >> >> > > > > > > > > > > > > > > > > > > > > Memory >> >> > > > > > > > > > > > > > > > > > > > > > > > Configuration for >> >> > > > TaskExecutors"[1], >> >> > > > > > > where >> >> > > > > > > > we >> >> > > > > > > > > > > > > describe >> >> > > > > > > > > > > > > > > how >> >> > > > > > > > > > > > > > > > to >> >> > > > > > > > > > > > > > > > > > > > improve >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory >> >> > > configurations. >> >> > > > > The >> >> > > > > > > > FLIP >> >> > > > > > > > > > > > document >> >> > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > mostly >> >> > > > > > > > > > > > > > > > > > > > based >> >> > > > > > > > > > > > > > > > > > > > > > on >> >> > > > > > > > > > > > > > > > > > > > > > > an >> >> > > > > > > > > > > > > > > > > > > > > > > > early design "Memory >> >> Management >> >> > > and >> >> > > > > > > > > > Configuration >> >> > > > > > > > > > > > > > > > > Reloaded"[2] >> >> > > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, >> >> > > > > > > > > > > > > > > > > > > > > > > > with updates from >> follow-up >> >> > > > > discussions >> >> > > > > > > > both >> >> > > > > > > > > > > online >> >> > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > > offline. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses >> several >> >> > > > > > shortcomings >> >> > > > > > > of >> >> > > > > > > > > > > current >> >> > > > > > > > > > > > > > > (Flink >> >> > > > > > > > > > > > > > > > > 1.9) >> >> > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory >> >> > > configuration. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Different >> configuration >> >> > for >> >> > > > > > > Streaming >> >> > > > > > > > > and >> >> > > > > > > > > > > > Batch. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complex and >> difficult >> >> > > > > > configuration >> >> > > > > > > of >> >> > > > > > > > > > > RocksDB >> >> > > > > > > > > > > > > in >> >> > > > > > > > > > > > > > > > > > Streaming. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Complicated, >> uncertain >> >> and >> >> > > > hard >> >> > > > > to >> >> > > > > > > > > > > understand. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve the >> >> > problems >> >> > > > can >> >> > > > > > be >> >> > > > > > > > > > > summarized >> >> > > > > > > > > > > > > as >> >> > > > > > > > > > > > > > > > > follows. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > - Extend memory >> manager >> >> to >> >> > > also >> >> > > > > > > account >> >> > > > > > > > > for >> >> > > > > > > > > > > > memory >> >> > > > > > > > > > > > > > > usage >> >> > > > > > > > > > > > > > > > > by >> >> > > > > > > > > > > > > > > > > > > > state >> >> > > > > > > > > > > > > > > > > > > > > > > > backends. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Modify how >> TaskExecutor >> >> > > memory >> >> > > > > is >> >> > > > > > > > > > > partitioned >> >> > > > > > > > > > > > > > > > accounted >> >> > > > > > > > > > > > > > > > > > > > > individual >> >> > > > > > > > > > > > > > > > > > > > > > > > memory reservations >> and >> >> > pools. >> >> > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory >> >> > > configuration >> >> > > > > > > options >> >> > > > > > > > > and >> >> > > > > > > > > > > > > > > calculations >> >> > > > > > > > > > > > > > > > > > > logics. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please find more details >> in >> >> the >> >> > > > FLIP >> >> > > > > > wiki >> >> > > > > > > > > > > document >> >> > > > > > > > > > > > > [1]. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > (Please note that the >> early >> >> > > design >> >> > > > > doc >> >> > > > > > > [2] >> >> > > > > > > > is >> >> > > > > > > > > > out >> >> > > > > > > > > > > > of >> >> > > > > > > > > > > > > > > sync, >> >> > > > > > > > > > > > > > > > > and >> >> > > > > > > > > > > > > > > > > > it >> >> > > > > > > > > > > > > > > > > > > > is >> >> > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the >> >> > > discussion >> >> > > > in >> >> > > > > > > this >> >> > > > > > > > > > > mailing >> >> > > > > > > > > > > > > list >> >> > > > > > > > > > > > > > > > > > thread.) >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your >> >> > > feedbacks. >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank you~ >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong Song >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing >> >> > > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> >
