I see. Under the assumption of strict determinism that should work. The original proposal had this point "don't compute inside the TM, compute outside and supply a full config", because that sounded more intuitive.
On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann <trohrm...@apache.org> wrote: > My understanding was that before starting the Flink process we call a > utility which calculates these values. I assume that this utility will do > the calculation based on a set of configured values (process memory, flink > memory, network memory etc.). Assuming that these values don't differ from > the values with which the JVM is started, it should be possible to > recompute them in the Flink process in order to set the values. > > > > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen <se...@apache.org> wrote: > > > When computing the values in the JVM process after it started, how would > > you deal with values like Max Direct Memory, Metaspace size. native > memory > > reservation (reduce heap size), etc? All the values that are parameters > to > > the JVM process and that need to be supplied at process startup? > > > > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > > > Thanks for the clarification. I have some more comments: > > > > > > - I would actually split the logic to compute the process memory > > > requirements and storing the values into two things. E.g. one could > name > > > the former TaskExecutorProcessUtility and the latter > > > TaskExecutorProcessMemory. But we can discuss this on the PR since it's > > > just a naming detail. > > > > > > - Generally, I'm not opposed to making configuration values overridable > > by > > > ENV variables. I think this is a very good idea and makes the > > > configurability of Flink processes easier. However, I think that adding > > > this functionality should not be part of this FLIP because it would > > simply > > > widen the scope unnecessarily. > > > > > > The reasons why I believe it is unnecessary are the following: For Yarn > > we > > > already create write a flink-conf.yaml which could be populated with > the > > > memory settings. For the other processes it should not make a > difference > > > whether the loaded Configuration is populated with the memory settings > > from > > > ENV variables or by using TaskExecutorProcessUtility to compute the > > missing > > > values from the loaded configuration. If the latter would not be > possible > > > (wrong or missing configuration values), then we should not have been > > able > > > to actually start the process in the first place. > > > > > > - Concerning the memory reservation: I agree with you that we need the > > > memory reservation functionality to make streaming jobs work with > > "managed" > > > memory. However, w/o this functionality the whole Flip would already > > bring > > > a good amount of improvements to our users when running batch jobs. > > > Moreover, by keeping the scope smaller we can complete the FLIP faster. > > > Hence, I would propose to address the memory reservation functionality > > as a > > > follow up FLIP (which Yu is working on if I'm not mistaken). > > > > > > Cheers, > > > Till > > > > > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang <danrtsey...@gmail.com> > > wrote: > > > > > > > Just add my 2 cents. > > > > > > > > Using environment variables to override the configuration for > different > > > > taskmanagers is better. > > > > We do not need to generate dedicated flink-conf.yaml for all > > > taskmanagers. > > > > A common flink-conf.yam and different environment variables are > enough. > > > > By reducing the distributed cached files, it could make launching a > > > > taskmanager faster. > > > > > > > > Stephan gives a good suggestion that we could move the logic into > > > > "GlobalConfiguration.loadConfig()" method. > > > > Maybe the client could also benefit from this. Different users do not > > > have > > > > to export FLINK_CONF_DIR to update few config options. > > > > > > > > > > > > Best, > > > > Yang > > > > > > > > Stephan Ewen <se...@apache.org> 于2019年8月28日周三 上午1:21写道： > > > > > > > > > One note on the Environment Variables and Configuration discussion. > > > > > > > > > > My understanding is that passed ENV variables are added to the > > > > > configuration in the "GlobalConfiguration.loadConfig()" method (or > > > > > similar). > > > > > For all the code inside Flink, it looks like the data was in the > > config > > > > to > > > > > start with, just that the scripts that compute the variables can > pass > > > the > > > > > values to the process without actually needing to write a file. > > > > > > > > > > For example the "GlobalConfiguration.loadConfig()" method would > take > > > any > > > > > ENV variable prefixed with "flink" and add it as a config key. > > > > > "flink_taskmanager_memory_size=2g" would become > > > "taskmanager.memory.size: > > > > > 2g". > > > > > > > > > > > > > > > On Tue, Aug 27, 2019 at 4:05 PM Xintong Song < > tonysong...@gmail.com> > > > > > wrote: > > > > > > > > > > > Thanks for the comments, Till. > > > > > > > > > > > > I've also seen your comments on the wiki page, but let's keep the > > > > > > discussion here. > > > > > > > > > > > > - Regarding 'TaskExecutorSpecifics', how do you think about > naming > > it > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > - Regarding passing memory configurations into task executors, > I'm > > in > > > > > favor > > > > > > of do it via environment variables rather than configurations, > with > > > the > > > > > > following two reasons. > > > > > > - It is easier to keep the memory options once calculate not to > > be > > > > > > changed with environment variables rather than configurations. > > > > > > - I'm not sure whether we should write the configuration in > > startup > > > > > > scripts. Writing changes into the configuration files when > running > > > the > > > > > > startup scripts does not sounds right to me. Or we could make a > > copy > > > of > > > > > > configuration files per flink cluster, and make the task executor > > to > > > > load > > > > > > from the copy, and clean up the copy after the cluster is > shutdown, > > > > which > > > > > > is complicated. (I think this is also what Stephan means in his > > > comment > > > > > on > > > > > > the wiki page?) > > > > > > - Regarding reserving memory, I think this change should be > > included > > > in > > > > > > this FLIP. I think a big part of motivations of this FLIP is to > > unify > > > > > > memory configuration for streaming / batch and make it easy for > > > > > configuring > > > > > > rocksdb memory. If we don't support memory reservation, then > > > streaming > > > > > jobs > > > > > > cannot use managed memory (neither on-heap or off-heap), which > > makes > > > > this > > > > > > FLIP incomplete. > > > > > > - Regarding network memory, I think you are right. I think we > > > probably > > > > > > don't need to change network stack from using direct memory to > > using > > > > > unsafe > > > > > > native memory. Network memory size is deterministic, cannot be > > > reserved > > > > > as > > > > > > managed memory does, and cannot be overused. I think it also > works > > if > > > > we > > > > > > simply keep using direct memory for network and include it in jvm > > max > > > > > > direct memory size. > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann < > > trohrm...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > thanks for addressing the comments and adding a more detailed > > > > > > > implementation plan. I have a couple of comments concerning the > > > > > > > implementation plan: > > > > > > > > > > > > > > - The name `TaskExecutorSpecifics` is not really descriptive. > > > > Choosing > > > > > a > > > > > > > different name could help here. > > > > > > > - I'm not sure whether I would pass the memory configuration to > > the > > > > > > > TaskExecutor via environment variables. I think it would be > > better > > > to > > > > > > write > > > > > > > it into the configuration one uses to start the TM process. > > > > > > > - If possible, I would exclude the memory reservation from this > > > FLIP > > > > > and > > > > > > > add this as part of a dedicated FLIP. > > > > > > > - If possible, then I would exclude changes to the network > stack > > > from > > > > > > this > > > > > > > FLIP. Maybe we can simply say that the direct memory needed by > > the > > > > > > network > > > > > > > stack is the framework direct memory requirement. Changing how > > the > > > > > memory > > > > > > > is allocated can happen in a second step. This would keep the > > scope > > > > of > > > > > > this > > > > > > > FLIP smaller. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM Xintong Song < > > > tonysong...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > I just updated the FLIP document on wiki [1], with the > > following > > > > > > changes. > > > > > > > > > > > > > > > > - Removed open question regarding MemorySegment > allocation. > > As > > > > > > > > discussed, we exclude this topic from the scope of this > > FLIP. > > > > > > > > - Updated content about JVM direct memory parameter > > according > > > to > > > > > > > recent > > > > > > > > discussions, and moved the other options to "Rejected > > > > > Alternatives" > > > > > > > for > > > > > > > > the > > > > > > > > moment. > > > > > > > > - Added implementation steps. > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM Stephan Ewen < > se...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > @Xintong: Concerning "wait for memory users before task > > dispose > > > > and > > > > > > > > memory > > > > > > > > > release": I agree, that's how it should be. Let's try it > out. > > > > > > > > > > > > > > > > > > @Xintong @Jingsong: Concerning " JVM does not wait for GC > > when > > > > > > > allocating > > > > > > > > > direct memory buffer": There seems to be pretty elaborate > > logic > > > > to > > > > > > free > > > > > > > > > buffers when allocating new ones. See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > > > > > > > > > > > @Till: Maybe. If we assume that the JVM default works (like > > > going > > > > > > with > > > > > > > > > option 2 and not setting "-XX:MaxDirectMemorySize" at all), > > > then > > > > I > > > > > > > think > > > > > > > > it > > > > > > > > > should be okay to set "-XX:MaxDirectMemorySize" to > > > > > > > > > "off_heap_managed_memory + direct_memory" even if we use > > > RocksDB. > > > > > > That > > > > > > > > is a > > > > > > > > > big if, though, I honestly have no idea :D Would be good to > > > > > > understand > > > > > > > > > this, though, because this would affect option (2) and > option > > > > > (1.2). > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 PM Xintong Song < > > > > > tonysong...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks for the inputs, Jingsong. > > > > > > > > > > > > > > > > > > > > Let me try to summarize your points. Please correct me if > > I'm > > > > > > wrong. > > > > > > > > > > > > > > > > > > > > - Memory consumers should always avoid returning > memory > > > > > segments > > > > > > > to > > > > > > > > > > memory manager while there are still un-cleaned > > > structures / > > > > > > > threads > > > > > > > > > > that > > > > > > > > > > may use the memory. Otherwise, it would cause serious > > > > problems > > > > > > by > > > > > > > > > having > > > > > > > > > > multiple consumers trying to use the same memory > > segment. > > > > > > > > > > - JVM does not wait for GC when allocating direct > memory > > > > > buffer. > > > > > > > > > > Therefore even we set proper max direct memory size > > limit, > > > > we > > > > > > may > > > > > > > > > still > > > > > > > > > > encounter direct memory oom if the GC cleaning memory > > > slower > > > > > > than > > > > > > > > the > > > > > > > > > > direct memory allocation. > > > > > > > > > > > > > > > > > > > > Am I understanding this correctly? > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 PM JingsongLee < > > > > > > lzljs3620...@aliyun.com > > > > > > > > > > .invalid> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi stephan: > > > > > > > > > > > > > > > > > > > > > > About option 2: > > > > > > > > > > > > > > > > > > > > > > if additional threads not cleanly shut down before we > can > > > > exit > > > > > > the > > > > > > > > > task: > > > > > > > > > > > In the current case of memory reuse, it has freed up > the > > > > memory > > > > > > it > > > > > > > > > > > uses. If this memory is used by other tasks and > > > asynchronous > > > > > > > threads > > > > > > > > > > > of exited task may still be writing, there will be > > > > concurrent > > > > > > > > security > > > > > > > > > > > problems, and even lead to errors in user computing > > > results. > > > > > > > > > > > > > > > > > > > > > > So I think this is a serious and intolerable bug, No > > matter > > > > > what > > > > > > > the > > > > > > > > > > > option is, it should be avoided. > > > > > > > > > > > > > > > > > > > > > > About direct memory cleaned by GC: > > > > > > > > > > > I don't think it is a good idea, I've encountered so > many > > > > > > > situations > > > > > > > > > > > that it's too late for GC to cause DirectMemory OOM. > > > Release > > > > > and > > > > > > > > > > > allocate DirectMemory depend on the type of user job, > > > which > > > > is > > > > > > > > > > > often beyond our control. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Jingsong Lee > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > From:Stephan Ewen <se...@apache.org> > > > > > > > > > > > Send Time:2019年8月19日(星期一) 15:56 > > > > > > > > > > > To:dev <dev@flink.apache.org> > > > > > > > > > > > Subject:Re: [DISCUSS] FLIP-49: Unified Memory > > Configuration > > > > for > > > > > > > > > > > TaskExecutors > > > > > > > > > > > > > > > > > > > > > > My main concern with option 2 (manually release memory) > > is > > > > that > > > > > > > > > segfaults > > > > > > > > > > > in the JVM send off all sorts of alarms on user ends. > So > > we > > > > > need > > > > > > to > > > > > > > > > > > guarantee that this never happens. > > > > > > > > > > > > > > > > > > > > > > The trickyness is in tasks that uses data structures / > > > > > algorithms > > > > > > > > with > > > > > > > > > > > additional threads, like hash table spill/read and > > sorting > > > > > > threads. > > > > > > > > We > > > > > > > > > > need > > > > > > > > > > > to ensure that these cleanly shut down before we can > exit > > > the > > > > > > task. > > > > > > > > > > > I am not sure that we have that guaranteed already, > > that's > > > > why > > > > > > > option > > > > > > > > > 1.1 > > > > > > > > > > > seemed simpler to me. > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2019 at 3:42 PM Xintong Song < > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for the comments, Stephan. Summarized in this > > way > > > > > really > > > > > > > > makes > > > > > > > > > > > > things easier to understand. > > > > > > > > > > > > > > > > > > > > > > > > I'm in favor of option 2, at least for the moment. I > > > think > > > > it > > > > > > is > > > > > > > > not > > > > > > > > > > that > > > > > > > > > > > > difficult to keep it segfault safe for memory > manager, > > as > > > > > long > > > > > > as > > > > > > > > we > > > > > > > > > > > always > > > > > > > > > > > > de-allocate the memory segment when it is released > from > > > the > > > > > > > memory > > > > > > > > > > > > consumers. Only if the memory consumer continue using > > the > > > > > > buffer > > > > > > > of > > > > > > > > > > > memory > > > > > > > > > > > > segment after releasing it, in which case we do want > > the > > > > job > > > > > to > > > > > > > > fail > > > > > > > > > so > > > > > > > > > > > we > > > > > > > > > > > > detect the memory leak early. > > > > > > > > > > > > > > > > > > > > > > > > For option 1.2, I don't think this is a good idea. > Not > > > only > > > > > > > because > > > > > > > > > the > > > > > > > > > > > > assumption (regular GC is enough to clean direct > > buffers) > > > > may > > > > > > not > > > > > > > > > > always > > > > > > > > > > > be > > > > > > > > > > > > true, but also it makes harder for finding problems > in > > > > cases > > > > > of > > > > > > > > > memory > > > > > > > > > > > > overuse. E.g., user configured some direct memory for > > the > > > > > user > > > > > > > > > > libraries. > > > > > > > > > > > > If the library actually use more direct memory then > > > > > configured, > > > > > > > > which > > > > > > > > > > > > cannot be cleaned by GC because they are still in > use, > > > may > > > > > lead > > > > > > > to > > > > > > > > > > > overuse > > > > > > > > > > > > of the total container memory. In that case, if it > > didn't > > > > > touch > > > > > > > the > > > > > > > > > JVM > > > > > > > > > > > > default max direct memory limit, we cannot get a > direct > > > > > memory > > > > > > > OOM > > > > > > > > > and > > > > > > > > > > it > > > > > > > > > > > > will become super hard to understand which part of > the > > > > > > > > configuration > > > > > > > > > > need > > > > > > > > > > > > to be updated. > > > > > > > > > > > > > > > > > > > > > > > > For option 1.1, it has the similar problem as 1.2, if > > the > > > > > > > exceeded > > > > > > > > > > direct > > > > > > > > > > > > memory does not reach the max direct memory limit > > > specified > > > > > by > > > > > > > the > > > > > > > > > > > > dedicated parameter. I think it is slightly better > than > > > > 1.2, > > > > > > only > > > > > > > > > > because > > > > > > > > > > > > we can tune the parameter. > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen < > > > > > se...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > About the "-XX:MaxDirectMemorySize" discussion, > maybe > > > let > > > > > me > > > > > > > > > > summarize > > > > > > > > > > > > it a > > > > > > > > > > > > > bit differently: > > > > > > > > > > > > > > > > > > > > > > > > > > We have the following two options: > > > > > > > > > > > > > > > > > > > > > > > > > > (1) We let MemorySegments be de-allocated by the > GC. > > > That > > > > > > makes > > > > > > > > it > > > > > > > > > > > > segfault > > > > > > > > > > > > > safe. But then we need a way to trigger GC in case > > > > > > > de-allocation > > > > > > > > > and > > > > > > > > > > > > > re-allocation of a bunch of segments happens > quickly, > > > > which > > > > > > is > > > > > > > > > often > > > > > > > > > > > the > > > > > > > > > > > > > case during batch scheduling or task restart. > > > > > > > > > > > > > - The "-XX:MaxDirectMemorySize" (option 1.1) is > one > > > way > > > > > to > > > > > > do > > > > > > > > > this > > > > > > > > > > > > > - Another way could be to have a dedicated > > > bookkeeping > > > > in > > > > > > the > > > > > > > > > > > > > MemoryManager (option 1.2), so that this is a > number > > > > > > > independent > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" parameter. > > > > > > > > > > > > > > > > > > > > > > > > > > (2) We manually allocate and de-allocate the memory > > for > > > > the > > > > > > > > > > > > MemorySegments > > > > > > > > > > > > > (option 2). That way we need not worry about > > triggering > > > > GC > > > > > by > > > > > > > > some > > > > > > > > > > > > > threshold or bookkeeping, but it is harder to > prevent > > > > > > > segfaults. > > > > > > > > We > > > > > > > > > > > need > > > > > > > > > > > > to > > > > > > > > > > > > > be very careful about when we release the memory > > > segments > > > > > > (only > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > cleanup phase of the main thread). > > > > > > > > > > > > > > > > > > > > > > > > > > If we go with option 1.1, we probably need to set > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" to > > "off_heap_managed_memory + > > > > > > > > > > direct_memory" > > > > > > > > > > > > and > > > > > > > > > > > > > have "direct_memory" as a separate reserved memory > > > pool. > > > > > > > Because > > > > > > > > if > > > > > > > > > > we > > > > > > > > > > > > just > > > > > > > > > > > > > set "-XX:MaxDirectMemorySize" to > > > > "off_heap_managed_memory + > > > > > > > > > > > > jvm_overhead", > > > > > > > > > > > > > then there will be times when that entire memory is > > > > > allocated > > > > > > > by > > > > > > > > > > direct > > > > > > > > > > > > > buffers and we have nothing left for the JVM > > overhead. > > > So > > > > > we > > > > > > > > either > > > > > > > > > > > need > > > > > > > > > > > > a > > > > > > > > > > > > > way to compensate for that (again some safety > margin > > > > cutoff > > > > > > > > value) > > > > > > > > > or > > > > > > > > > > > we > > > > > > > > > > > > > will exceed container memory. > > > > > > > > > > > > > > > > > > > > > > > > > > If we go with option 1.2, we need to be aware that > it > > > > takes > > > > > > > > > elaborate > > > > > > > > > > > > logic > > > > > > > > > > > > > to push recycling of direct buffers without always > > > > > > triggering a > > > > > > > > > full > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My first guess is that the options will be easiest > to > > > do > > > > in > > > > > > the > > > > > > > > > > > following > > > > > > > > > > > > > order: > > > > > > > > > > > > > > > > > > > > > > > > > > - Option 1.1 with a dedicated direct_memory > > > parameter, > > > > as > > > > > > > > > discussed > > > > > > > > > > > > > above. We would need to find a way to set the > > > > direct_memory > > > > > > > > > parameter > > > > > > > > > > > by > > > > > > > > > > > > > default. We could start with 64 MB and see how it > > goes > > > in > > > > > > > > practice. > > > > > > > > > > One > > > > > > > > > > > > > danger I see is that setting this loo low can > cause a > > > > bunch > > > > > > of > > > > > > > > > > > additional > > > > > > > > > > > > > GCs compared to before (we need to watch this > > > carefully). > > > > > > > > > > > > > > > > > > > > > > > > > > - Option 2. It is actually quite simple to > > implement, > > > > we > > > > > > > could > > > > > > > > > try > > > > > > > > > > > how > > > > > > > > > > > > > segfault safe we are at the moment. > > > > > > > > > > > > > > > > > > > > > > > > > > - Option 1.2: We would not touch the > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > > parameter > > > > > > > > > > > > > at all and assume that all the direct memory > > > allocations > > > > > that > > > > > > > the > > > > > > > > > JVM > > > > > > > > > > > and > > > > > > > > > > > > > Netty do are infrequent enough to be cleaned up > fast > > > > enough > > > > > > > > through > > > > > > > > > > > > regular > > > > > > > > > > > > > GC. I am not sure if that is a valid assumption, > > > though. > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song < > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your opinion Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was > wondering > > > > > whether > > > > > > > we > > > > > > > > > can > > > > > > > > > > > > avoid > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap managed > memory > > > and > > > > > > > network > > > > > > > > > > > memory > > > > > > > > > > > > > with > > > > > > > > > > > > > > alternative 3. But after giving it a second > > thought, > > > I > > > > > > think > > > > > > > > even > > > > > > > > > > for > > > > > > > > > > > > > > alternative 3 using direct memory for off-heap > > > managed > > > > > > memory > > > > > > > > > could > > > > > > > > > > > > cause > > > > > > > > > > > > > > problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding your concern, I think what proposed in > > this > > > > > FLIP > > > > > > it > > > > > > > > to > > > > > > > > > > have > > > > > > > > > > > > > both > > > > > > > > > > > > > > off-heap managed memory and network memory > > allocated > > > > > > through > > > > > > > > > > > > > > Unsafe.allocate(), which means they are > practically > > > > > native > > > > > > > > memory > > > > > > > > > > and > > > > > > > > > > > > not > > > > > > > > > > > > > > limited by JVM max direct memory. The only parts > of > > > > > memory > > > > > > > > > limited > > > > > > > > > > by > > > > > > > > > > > > JVM > > > > > > > > > > > > > > max direct memory are task off-heap memory and > JVM > > > > > > overhead, > > > > > > > > > which > > > > > > > > > > > are > > > > > > > > > > > > > > exactly alternative 2 suggests to set the JVM max > > > > direct > > > > > > > memory > > > > > > > > > to. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till Rohrmann < > > > > > > > > > > trohrm...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I > > understand > > > > the > > > > > > two > > > > > > > > > > > > alternatives > > > > > > > > > > > > > > > now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would be in favour of option 2 because it > makes > > > > > things > > > > > > > > > > explicit. > > > > > > > > > > > If > > > > > > > > > > > > > we > > > > > > > > > > > > > > > don't limit the direct memory, I fear that we > > might > > > > end > > > > > > up > > > > > > > > in a > > > > > > > > > > > > similar > > > > > > > > > > > > > > > situation as we are currently in: The user > might > > > see > > > > > that > > > > > > > her > > > > > > > > > > > process > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > killed by the OS and does not know why this is > > the > > > > > case. > > > > > > > > > > > > Consequently, > > > > > > > > > > > > > > she > > > > > > > > > > > > > > > tries to decrease the process memory size > > (similar > > > to > > > > > > > > > increasing > > > > > > > > > > > the > > > > > > > > > > > > > > cutoff > > > > > > > > > > > > > > > ratio) in order to accommodate for the extra > > direct > > > > > > memory. > > > > > > > > > Even > > > > > > > > > > > > worse, > > > > > > > > > > > > > > she > > > > > > > > > > > > > > > tries to decrease memory budgets which are not > > > fully > > > > > used > > > > > > > and > > > > > > > > > > hence > > > > > > > > > > > > > won't > > > > > > > > > > > > > > > change the overall memory consumption. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong Song < > > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me explain this with a concrete example > > Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let's say we have the following scenario. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Total Process Memory: 1GB > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap Memory + JVM > > > > > > Overhead): > > > > > > > > > 200MB > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM Metaspace, > > > > > Off-Heap > > > > > > > > > Managed > > > > > > > > > > > > Memory > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > Network Memory): 800MB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For alternative 2, we set > > -XX:MaxDirectMemorySize > > > > to > > > > > > > 200MB. > > > > > > > > > > > > > > > > For alternative 3, we set > > -XX:MaxDirectMemorySize > > > > to > > > > > a > > > > > > > very > > > > > > > > > > large > > > > > > > > > > > > > > value, > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the actual direct memory usage of Task > > > Off-Heap > > > > > > Memory > > > > > > > > and > > > > > > > > > > JVM > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > > > > > > do not exceed 200MB, then alternative 2 and > > > > > > alternative 3 > > > > > > > > > > should > > > > > > > > > > > > have > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > same utility. Setting larger > > > > -XX:MaxDirectMemorySize > > > > > > will > > > > > > > > not > > > > > > > > > > > > reduce > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > sizes of the other memory pools. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the actual direct memory usage of Task > > > Off-Heap > > > > > > Memory > > > > > > > > and > > > > > > > > > > JVM > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, then > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Alternative 2 suffers from frequent OOM. > > To > > > > > avoid > > > > > > > > that, > > > > > > > > > > the > > > > > > > > > > > > only > > > > > > > > > > > > > > > thing > > > > > > > > > > > > > > > > user can do is to modify the configuration > > and > > > > > > > increase > > > > > > > > > JVM > > > > > > > > > > > > Direct > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM Overhead). > Let's > > > say > > > > > > that > > > > > > > > user > > > > > > > > > > > > > increases > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will reduce > the > > > > total > > > > > > > size > > > > > > > > of > > > > > > > > > > > other > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > pools to 750MB, given the total process > > memory > > > > > > remains > > > > > > > > > 1GB. > > > > > > > > > > > > > > > > - For alternative 3, there is no chance of > > > > direct > > > > > > OOM. > > > > > > > > > There > > > > > > > > > > > are > > > > > > > > > > > > > > > chances > > > > > > > > > > > > > > > > of exceeding the total process memory > limit, > > > but > > > > > > given > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > not use up all the reserved native memory > > > > > (Off-Heap > > > > > > > > > Managed > > > > > > > > > > > > > Memory, > > > > > > > > > > > > > > > > Network > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > > is > > > > > > > > > > > > > > slightly > > > > > > > > > > > > > > > > above > > > > > > > > > > > > > > > > yet very close to 200MB, user probably do > > not > > > > need > > > > > > to > > > > > > > > > change > > > > > > > > > > > the > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, I think from the user's > > perspective, a > > > > > > > feasible > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > for alternative 2 may lead to lower resource > > > > > > utilization > > > > > > > > > > compared > > > > > > > > > > > > to > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till > Rohrmann > > < > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I guess you have to help me understand the > > > > > difference > > > > > > > > > between > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > > > > > > > > > and 3 wrt to memory under utilization > > Xintong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Alternative 2: set XX:MaxDirectMemorySize > > to > > > > Task > > > > > > > > > Off-Heap > > > > > > > > > > > > Memory > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > > > > > > Overhead. Then there is the risk that this > > size > > > > is > > > > > > too > > > > > > > > low > > > > > > > > > > > > > resulting > > > > > > > > > > > > > > > in a > > > > > > > > > > > > > > > > > lot of garbage collection and potentially > an > > > OOM. > > > > > > > > > > > > > > > > > - Alternative 3: set XX:MaxDirectMemorySize > > to > > > > > > > something > > > > > > > > > > larger > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > alternative 2. This would of course reduce > > the > > > > > sizes > > > > > > of > > > > > > > > the > > > > > > > > > > > other > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How would alternative 2 now result in an > > under > > > > > > > > utilization > > > > > > > > > of > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > compared to alternative 3? If alternative 3 > > > > > strictly > > > > > > > > sets a > > > > > > > > > > > > higher > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > direct memory size and we use only little, > > > then I > > > > > > would > > > > > > > > > > expect > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > alternative 3 results in memory under > > > > utilization. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang Wang < > > > > > > > > > > > danrtsey...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi xintong,till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Native and Direct Memory > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My point is setting a very large max > direct > > > > > memory > > > > > > > size > > > > > > > > > > when > > > > > > > > > > > we > > > > > > > > > > > > > do > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > differentiate direct and native memory. > If > > > the > > > > > > direct > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > direct memory and framework direct > > > memory,could > > > > > be > > > > > > > > > > calculated > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > > > > > > > > i am in favor of setting direct memory > with > > > > fixed > > > > > > > > value. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Memory Calculation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn and k8s,we > > > need > > > > to > > > > > > > check > > > > > > > > > the > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > configurations in client to avoid > > submitting > > > > > > > > successfully > > > > > > > > > > and > > > > > > > > > > > > > > failing > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > the flink master. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com > > > > >于2019年8月13日 > > > > > > > > > 周二22:07写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About MemorySegment, I think you are > > right > > > > that > > > > > > we > > > > > > > > > should > > > > > > > > > > > not > > > > > > > > > > > > > > > include > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > issue in the scope of this FLIP. This > > FLIP > > > > > should > > > > > > > > > > > concentrate > > > > > > > > > > > > > on > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > configure memory pools for > TaskExecutors, > > > > with > > > > > > > > minimum > > > > > > > > > > > > > > involvement > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > memory consumers use it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About direct memory, I think > alternative > > 3 > > > > may > > > > > > not > > > > > > > > > having > > > > > > > > > > > the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > reservation issue that alternative 2 > > does, > > > > but > > > > > at > > > > > > > the > > > > > > > > > > cost > > > > > > > > > > > of > > > > > > > > > > > > > > risk > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > using memory at the container level, > > which > > > is > > > > > not > > > > > > > > good. > > > > > > > > > > My > > > > > > > > > > > > > point > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and "JVM > > > > Overhead" > > > > > > are > > > > > > > > not > > > > > > > > > > easy > > > > > > > > > > > > to > > > > > > > > > > > > > > > > config. > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > > > > > > > > > alternative 2, users might configure > them > > > > > higher > > > > > > > than > > > > > > > > > > what > > > > > > > > > > > > > > actually > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > > > > > > > > > just to avoid getting a direct OOM. For > > > > > > alternative > > > > > > > > 3, > > > > > > > > > > > users > > > > > > > > > > > > do > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > > > direct OOM, so they may not config the > > two > > > > > > options > > > > > > > > > > > > aggressively > > > > > > > > > > > > > > > high. > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > the consequences are risks of overall > > > > container > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM Till > > > > Rohrmann < > > > > > > > > > > > > > > > trohrm...@apache.org> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for proposing this FLIP > Xintong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All in all I think it already looks > > quite > > > > > good. > > > > > > > > > > > Concerning > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > > > > > > > > > > question about allocating memory > > > segments, > > > > I > > > > > > was > > > > > > > > > > > wondering > > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > strictly necessary to do in the > context > > > of > > > > > this > > > > > > > > FLIP > > > > > > > > > or > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > > be done as a follow up? Without > knowing > > > all > > > > > > > > details, > > > > > > > > > I > > > > > > > > > > > > would > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > > > > > > > > > > that we would widen the scope of this > > > FLIP > > > > > too > > > > > > > much > > > > > > > > > > > because > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > to touch all the existing call sites > of > > > the > > > > > > > > > > MemoryManager > > > > > > > > > > > > > where > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > > > > > > > > > > memory segments (this should mainly > be > > > > batch > > > > > > > > > > operators). > > > > > > > > > > > > The > > > > > > > > > > > > > > > > addition > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > the memory reservation call to the > > > > > > MemoryManager > > > > > > > > > should > > > > > > > > > > > not > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > this and I would hope that this is > the > > > only > > > > > > point > > > > > > > > of > > > > > > > > > > > > > > interaction > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > streaming job would have with the > > > > > > MemoryManager. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Concerning the second open question > > about > > > > > > setting > > > > > > > > or > > > > > > > > > > not > > > > > > > > > > > > > > setting > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > > > > direct memory limit, I would also be > > > > > interested > > > > > > > why > > > > > > > > > > Yang > > > > > > > > > > > > Wang > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > > > > > > > > > > leaving it open would be best. My > > concern > > > > > about > > > > > > > > this > > > > > > > > > > > would > > > > > > > > > > > > be > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > be in a similar situation as we are > now > > > > with > > > > > > the > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > > > the different memory pools are not > > > clearly > > > > > > > > separated > > > > > > > > > > and > > > > > > > > > > > > can > > > > > > > > > > > > > > > spill > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > a different pool, then it is quite > hard > > > to > > > > > > > > understand > > > > > > > > > > > what > > > > > > > > > > > > > > > exactly > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > > > > > > > > > > process to get killed for using too > > much > > > > > > memory. > > > > > > > > This > > > > > > > > > > > could > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > > > > > > > > > > lead to a similar situation what we > > have > > > > with > > > > > > the > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > setting a sane default value for max > > > direct > > > > > > > memory > > > > > > > > > and > > > > > > > > > > > > giving > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > option to increase it if he runs into > > an > > > > OOM. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Xintong, how would alternative 2 > lead > > to > > > > > lower > > > > > > > > > memory > > > > > > > > > > > > > > > utilization > > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > alternative 3 where we set the direct > > > > memory > > > > > > to a > > > > > > > > > > higher > > > > > > > > > > > > > value? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM > Xintong > > > > Song < > > > > > > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, Yang. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding your comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* > > > > > > > > > > > > > > > > > > > > > I think setting a very large max > > direct > > > > > > memory > > > > > > > > size > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not worry > > about > > > > > > direct > > > > > > > > OOM, > > > > > > > > > > and > > > > > > > > > > > > we > > > > > > > > > > > > > > > don't > > > > > > > > > > > > > > > > > even > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > > > > > to allocate managed / network > memory > > > with > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > > > > > > > > > > > > > However, there are also some down > > sides > > > > of > > > > > > > doing > > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I can think of is > that > > > if > > > > a > > > > > > task > > > > > > > > > > > executor > > > > > > > > > > > > > > > > container > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > killed due to overusing memory, > it > > > > could > > > > > > be > > > > > > > > hard > > > > > > > > > > for > > > > > > > > > > > > use > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > of the memory is overused. > > > > > > > > > > > > > > > > > > > > > - Another down side is that the > > JVM > > > > > never > > > > > > > > > trigger > > > > > > > > > > GC > > > > > > > > > > > > due > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > > > > > direct memory limit, because the > > > limit > > > > > is > > > > > > > too > > > > > > > > > high > > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > > > > > > > > > > > means we kind of relay on heap > > > memory > > > > to > > > > > > > > trigger > > > > > > > > > > GC > > > > > > > > > > > > and > > > > > > > > > > > > > > > > release > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > > > > > > > > > > memory. That could be a problem > in > > > > cases > > > > > > > where > > > > > > > > > we > > > > > > > > > > > have > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > > > > usage but not enough heap > activity > > > to > > > > > > > trigger > > > > > > > > > the > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe you can share your reasons > for > > > > > > preferring > > > > > > > > > > > setting a > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > > > > > > > > > > > if there are anything else I > > > overlooked. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* > > > > > > > > > > > > > > > > > > > > > If there is any conflict between > > > multiple > > > > > > > > > > configuration > > > > > > > > > > > > > that > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > > > > explicitly specified, I think we > > should > > > > > throw > > > > > > > an > > > > > > > > > > error. > > > > > > > > > > > > > > > > > > > > > I think doing checking on the > client > > > side > > > > > is > > > > > > a > > > > > > > > good > > > > > > > > > > > idea, > > > > > > > > > > > > > so > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > > > > > > > > > > > K8s we can discover the problem > > before > > > > > > > submitting > > > > > > > > > the > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > is always a good thing. > > > > > > > > > > > > > > > > > > > > > But we can not only rely on the > > client > > > > side > > > > > > > > > checking, > > > > > > > > > > > > > because > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > standalone cluster TaskManagers on > > > > > different > > > > > > > > > machines > > > > > > > > > > > may > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > > > configurations and the client does > > see > > > > > that. > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 PM Yang > > > Wang > > > > < > > > > > > > > > > > > > > > danrtsey...@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi xintong, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed > proposal. > > > > After > > > > > > all > > > > > > > > the > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > introduced, it will be more > > powerful > > > to > > > > > > > control > > > > > > > > > the > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > > > > > > > > > > > > just have few questions about it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Native and Direct Memory > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We do not differentiate user > direct > > > > > memory > > > > > > > and > > > > > > > > > > native > > > > > > > > > > > > > > memory. > > > > > > > > > > > > > > > > > They > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > included in task off-heap memory. > > > > Right? > > > > > > So i > > > > > > > > > don’t > > > > > > > > > > > > think > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > > > > > > > > > > > > the -XX:MaxDirectMemorySize > > > properly. I > > > > > > > prefer > > > > > > > > > > > leaving > > > > > > > > > > > > > it a > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the sum of and fine-grained > > > > > > memory(network > > > > > > > > > > memory, > > > > > > > > > > > > > > managed > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > > > > > > > > > > > > is larger than total process > > memory, > > > > how > > > > > do > > > > > > > we > > > > > > > > > deal > > > > > > > > > > > > with > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > > > > > > > > > > > > we need to check the memory > > > > configuration > > > > > > in > > > > > > > > > > client? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song < > > tonysong...@gmail.com> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We would like to start a > > discussion > > > > > > thread > > > > > > > on > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > > > > > > > > > > > > Configuration for > > > TaskExecutors"[1], > > > > > > where > > > > > > > we > > > > > > > > > > > > describe > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > > configurations. > > > > The > > > > > > > FLIP > > > > > > > > > > > document > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > early design "Memory Management > > and > > > > > > > > > Configuration > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > > > > > > > > > > > > > with updates from follow-up > > > > discussions > > > > > > > both > > > > > > > > > > online > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses several > > > > > shortcomings > > > > > > of > > > > > > > > > > current > > > > > > > > > > > > > > (Flink > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > > configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Different configuration > for > > > > > > Streaming > > > > > > > > and > > > > > > > > > > > Batch. > > > > > > > > > > > > > > > > > > > > > > > - Complex and difficult > > > > > configuration > > > > > > of > > > > > > > > > > RocksDB > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > > > > > > > > > > > > > - Complicated, uncertain and > > > hard > > > > to > > > > > > > > > > understand. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve the > problems > > > can > > > > > be > > > > > > > > > > summarized > > > > > > > > > > > > as > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Extend memory manager to > > also > > > > > > account > > > > > > > > for > > > > > > > > > > > memory > > > > > > > > > > > > > > usage > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > > > > > > > > > > > > > - Modify how TaskExecutor > > memory > > > > is > > > > > > > > > > partitioned > > > > > > > > > > > > > > > accounted > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > > > > > > > > > > > > > memory reservations and > pools. > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory > > configuration > > > > > > options > > > > > > > > and > > > > > > > > > > > > > > calculations > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please find more details in the > > > FLIP > > > > > wiki > > > > > > > > > > document > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please note that the early > > design > > > > doc > > > > > > [2] > > > > > > > is > > > > > > > > > out > > > > > > > > > > > of > > > > > > > > > > > > > > sync, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the > > discussion > > > in > > > > > > this > > > > > > > > > > mailing > > > > > > > > > > > > list > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your > > feedbacks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song < > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your opinion Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm also in favor of alternative 2. I was > wondering > > > > > whether > > > > > > > we > > > > > > > > > can > > > > > > > > > > > > avoid > > > > > > > > > > > > > > using Unsafe.allocate() for off-heap managed > memory > > > and > > > > > > > network > > > > > > > > > > > memory > > > > > > > > > > > > > with > > > > > > > > > > > > > > alternative 3. But after giving it a second > > thought, > > > I > > > > > > think > > > > > > > > even > > > > > > > > > > for > > > > > > > > > > > > > > alternative 3 using direct memory for off-heap > > > managed > > > > > > memory > > > > > > > > > could > > > > > > > > > > > > cause > > > > > > > > > > > > > > problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding your concern, I think what proposed in > > this > > > > > FLIP > > > > > > it > > > > > > > > to > > > > > > > > > > have > > > > > > > > > > > > > both > > > > > > > > > > > > > > off-heap managed memory and network memory > > allocated > > > > > > through > > > > > > > > > > > > > > Unsafe.allocate(), which means they are > practically > > > > > native > > > > > > > > memory > > > > > > > > > > and > > > > > > > > > > > > not > > > > > > > > > > > > > > limited by JVM max direct memory. The only parts > of > > > > > memory > > > > > > > > > limited > > > > > > > > > > by > > > > > > > > > > > > JVM > > > > > > > > > > > > > > max direct memory are task off-heap memory and > JVM > > > > > > overhead, > > > > > > > > > which > > > > > > > > > > > are > > > > > > > > > > > > > > exactly alternative 2 suggests to set the JVM max > > > > direct > > > > > > > memory > > > > > > > > > to. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 1:48 PM Till Rohrmann < > > > > > > > > > > trohrm...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the clarification Xintong. I > > understand > > > > the > > > > > > two > > > > > > > > > > > > alternatives > > > > > > > > > > > > > > > now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would be in favour of option 2 because it > makes > > > > > things > > > > > > > > > > explicit. > > > > > > > > > > > If > > > > > > > > > > > > > we > > > > > > > > > > > > > > > don't limit the direct memory, I fear that we > > might > > > > end > > > > > > up > > > > > > > > in a > > > > > > > > > > > > similar > > > > > > > > > > > > > > > situation as we are currently in: The user > might > > > see > > > > > that > > > > > > > her > > > > > > > > > > > process > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > killed by the OS and does not know why this is > > the > > > > > case. > > > > > > > > > > > > Consequently, > > > > > > > > > > > > > > she > > > > > > > > > > > > > > > tries to decrease the process memory size > > (similar > > > to > > > > > > > > > increasing > > > > > > > > > > > the > > > > > > > > > > > > > > cutoff > > > > > > > > > > > > > > > ratio) in order to accommodate for the extra > > direct > > > > > > memory. > > > > > > > > > Even > > > > > > > > > > > > worse, > > > > > > > > > > > > > > she > > > > > > > > > > > > > > > tries to decrease memory budgets which are not > > > fully > > > > > used > > > > > > > and > > > > > > > > > > hence > > > > > > > > > > > > > won't > > > > > > > > > > > > > > > change the overall memory consumption. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong Song < > > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me explain this with a concrete example > > Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let's say we have the following scenario. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Total Process Memory: 1GB > > > > > > > > > > > > > > > > JVM Direct Memory (Task Off-Heap Memory + JVM > > > > > > Overhead): > > > > > > > > > 200MB > > > > > > > > > > > > > > > > Other Memory (JVM Heap Memory, JVM Metaspace, > > > > > Off-Heap > > > > > > > > > Managed > > > > > > > > > > > > Memory > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > Network Memory): 800MB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For alternative 2, we set > > -XX:MaxDirectMemorySize > > > > to > > > > > > > 200MB. > > > > > > > > > > > > > > > > For alternative 3, we set > > -XX:MaxDirectMemorySize > > > > to > > > > > a > > > > > > > very > > > > > > > > > > large > > > > > > > > > > > > > > value, > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the actual direct memory usage of Task > > > Off-Heap > > > > > > Memory > > > > > > > > and > > > > > > > > > > JVM > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > > > > > > do not exceed 200MB, then alternative 2 and > > > > > > alternative 3 > > > > > > > > > > should > > > > > > > > > > > > have > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > same utility. Setting larger > > > > -XX:MaxDirectMemorySize > > > > > > will > > > > > > > > not > > > > > > > > > > > > reduce > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > sizes of the other memory pools. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the actual direct memory usage of Task > > > Off-Heap > > > > > > Memory > > > > > > > > and > > > > > > > > > > JVM > > > > > > > > > > > > > > > > Overhead potentially exceed 200MB, then > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Alternative 2 suffers from frequent OOM. > > To > > > > > avoid > > > > > > > > that, > > > > > > > > > > the > > > > > > > > > > > > only > > > > > > > > > > > > > > > thing > > > > > > > > > > > > > > > > user can do is to modify the configuration > > and > > > > > > > increase > > > > > > > > > JVM > > > > > > > > > > > > Direct > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > > > > > (Task Off-Heap Memory + JVM Overhead). > Let's > > > say > > > > > > that > > > > > > > > user > > > > > > > > > > > > > increases > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > > > > > Direct Memory to 250MB, this will reduce > the > > > > total > > > > > > > size > > > > > > > > of > > > > > > > > > > > other > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > pools to 750MB, given the total process > > memory > > > > > > remains > > > > > > > > > 1GB. > > > > > > > > > > > > > > > > - For alternative 3, there is no chance of > > > > direct > > > > > > OOM. > > > > > > > > > There > > > > > > > > > > > are > > > > > > > > > > > > > > > chances > > > > > > > > > > > > > > > > of exceeding the total process memory > limit, > > > but > > > > > > given > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > not use up all the reserved native memory > > > > > (Off-Heap > > > > > > > > > Managed > > > > > > > > > > > > > Memory, > > > > > > > > > > > > > > > > Network > > > > > > > > > > > > > > > > Memory, JVM Metaspace), if the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > > is > > > > > > > > > > > > > > slightly > > > > > > > > > > > > > > > > above > > > > > > > > > > > > > > > > yet very close to 200MB, user probably do > > not > > > > need > > > > > > to > > > > > > > > > change > > > > > > > > > > > the > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, I think from the user's > > perspective, a > > > > > > > feasible > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > for alternative 2 may lead to lower resource > > > > > > utilization > > > > > > > > > > compared > > > > > > > > > > > > to > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till > Rohrmann > > < > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I guess you have to help me understand the > > > > > difference > > > > > > > > > between > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > > > > > > > > > and 3 wrt to memory under utilization > > Xintong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Alternative 2: set XX:MaxDirectMemorySize > > to > > > > Task > > > > > > > > > Off-Heap > > > > > > > > > > > > Memory > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > > > > > > Overhead. Then there is the risk that this > > size > > > > is > > > > > > too > > > > > > > > low > > > > > > > > > > > > > resulting > > > > > > > > > > > > > > > in a > > > > > > > > > > > > > > > > > lot of garbage collection and potentially > an > > > OOM. > > > > > > > > > > > > > > > > > - Alternative 3: set XX:MaxDirectMemorySize > > to > > > > > > > something > > > > > > > > > > larger > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > alternative 2. This would of course reduce > > the > > > > > sizes > > > > > > of > > > > > > > > the > > > > > > > > > > > other > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How would alternative 2 now result in an > > under > > > > > > > > utilization > > > > > > > > > of > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > compared to alternative 3? If alternative 3 > > > > > strictly > > > > > > > > sets a > > > > > > > > > > > > higher > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > direct memory size and we use only little, > > > then I > > > > > > would > > > > > > > > > > expect > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > alternative 3 results in memory under > > > > utilization. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang Wang < > > > > > > > > > > > danrtsey...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi xintong,till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Native and Direct Memory > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My point is setting a very large max > direct > > > > > memory > > > > > > > size > > > > > > > > > > when > > > > > > > > > > > we > > > > > > > > > > > > > do > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > differentiate direct and native memory. > If > > > the > > > > > > direct > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > direct memory and framework direct > > > memory,could > > > > > be > > > > > > > > > > calculated > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > > > > > > > > i am in favor of setting direct memory > with > > > > fixed > > > > > > > > value. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Memory Calculation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree with xintong. For Yarn and k8s,we > > > need > > > > to > > > > > > > check > > > > > > > > > the > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > configurations in client to avoid > > submitting > > > > > > > > successfully > > > > > > > > > > and > > > > > > > > > > > > > > failing > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > the flink master. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com > > > > >于2019年8月13日 > > > > > > > > > 周二22:07写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for replying, Till. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About MemorySegment, I think you are > > right > > > > that > > > > > > we > > > > > > > > > should > > > > > > > > > > > not > > > > > > > > > > > > > > > include > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > issue in the scope of this FLIP. This > > FLIP > > > > > should > > > > > > > > > > > concentrate > > > > > > > > > > > > > on > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > configure memory pools for > TaskExecutors, > > > > with > > > > > > > > minimum > > > > > > > > > > > > > > involvement > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > memory consumers use it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > About direct memory, I think > alternative > > 3 > > > > may > > > > > > not > > > > > > > > > having > > > > > > > > > > > the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > reservation issue that alternative 2 > > does, > > > > but > > > > > at > > > > > > > the > > > > > > > > > > cost > > > > > > > > > > > of > > > > > > > > > > > > > > risk > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > using memory at the container level, > > which > > > is > > > > > not > > > > > > > > good. > > > > > > > > > > My > > > > > > > > > > > > > point > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > both "Task Off-Heap Memory" and "JVM > > > > Overhead" > > > > > > are > > > > > > > > not > > > > > > > > > > easy > > > > > > > > > > > > to > > > > > > > > > > > > > > > > config. > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > > > > > > > > > alternative 2, users might configure > them > > > > > higher > > > > > > > than > > > > > > > > > > what > > > > > > > > > > > > > > actually > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > > > > > > > > > just to avoid getting a direct OOM. For > > > > > > alternative > > > > > > > > 3, > > > > > > > > > > > users > > > > > > > > > > > > do > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > > > direct OOM, so they may not config the > > two > > > > > > options > > > > > > > > > > > > aggressively > > > > > > > > > > > > > > > high. > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > the consequences are risks of overall > > > > container > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM Till > > > > Rohrmann < > > > > > > > > > > > > > > > trohrm...@apache.org> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for proposing this FLIP > Xintong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All in all I think it already looks > > quite > > > > > good. > > > > > > > > > > > Concerning > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > > > > > > > > > > question about allocating memory > > > segments, > > > > I > > > > > > was > > > > > > > > > > > wondering > > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > strictly necessary to do in the > context > > > of > > > > > this > > > > > > > > FLIP > > > > > > > > > or > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > > be done as a follow up? Without > knowing > > > all > > > > > > > > details, > > > > > > > > > I > > > > > > > > > > > > would > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > > > > > > > > > > that we would widen the scope of this > > > FLIP > > > > > too > > > > > > > much > > > > > > > > > > > because > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > to touch all the existing call sites > of > > > the > > > > > > > > > > MemoryManager > > > > > > > > > > > > > where > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > > > > > > > > > > memory segments (this should mainly > be > > > > batch > > > > > > > > > > operators). > > > > > > > > > > > > The > > > > > > > > > > > > > > > > addition > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > the memory reservation call to the > > > > > > MemoryManager > > > > > > > > > should > > > > > > > > > > > not > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > this and I would hope that this is > the > > > only > > > > > > point > > > > > > > > of > > > > > > > > > > > > > > interaction > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > streaming job would have with the > > > > > > MemoryManager. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Concerning the second open question > > about > > > > > > setting > > > > > > > > or > > > > > > > > > > not > > > > > > > > > > > > > > setting > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > > > > direct memory limit, I would also be > > > > > interested > > > > > > > why > > > > > > > > > > Yang > > > > > > > > > > > > Wang > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > > > > > > > > > > leaving it open would be best. My > > concern > > > > > about > > > > > > > > this > > > > > > > > > > > would > > > > > > > > > > > > be > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > be in a similar situation as we are > now > > > > with > > > > > > the > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > > > the different memory pools are not > > > clearly > > > > > > > > separated > > > > > > > > > > and > > > > > > > > > > > > can > > > > > > > > > > > > > > > spill > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > a different pool, then it is quite > hard > > > to > > > > > > > > understand > > > > > > > > > > > what > > > > > > > > > > > > > > > exactly > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > > > > > > > > > > process to get killed for using too > > much > > > > > > memory. > > > > > > > > This > > > > > > > > > > > could > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > > > > > > > > > > lead to a similar situation what we > > have > > > > with > > > > > > the > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > setting a sane default value for max > > > direct > > > > > > > memory > > > > > > > > > and > > > > > > > > > > > > giving > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > option to increase it if he runs into > > an > > > > OOM. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Xintong, how would alternative 2 > lead > > to > > > > > lower > > > > > > > > > memory > > > > > > > > > > > > > > > utilization > > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > alternative 3 where we set the direct > > > > memory > > > > > > to a > > > > > > > > > > higher > > > > > > > > > > > > > value? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM > Xintong > > > > Song < > > > > > > > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback, Yang. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding your comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Native and Direct Memory* > > > > > > > > > > > > > > > > > > > > > I think setting a very large max > > direct > > > > > > memory > > > > > > > > size > > > > > > > > > > > > > > definitely > > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > good sides. E.g., we do not worry > > about > > > > > > direct > > > > > > > > OOM, > > > > > > > > > > and > > > > > > > > > > > > we > > > > > > > > > > > > > > > don't > > > > > > > > > > > > > > > > > even > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > > > > > to allocate managed / network > memory > > > with > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > > > > > > > > > > > > > However, there are also some down > > sides > > > > of > > > > > > > doing > > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I can think of is > that > > > if > > > > a > > > > > > task > > > > > > > > > > > executor > > > > > > > > > > > > > > > > container > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > killed due to overusing memory, > it > > > > could > > > > > > be > > > > > > > > hard > > > > > > > > > > for > > > > > > > > > > > > use > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > of the memory is overused. > > > > > > > > > > > > > > > > > > > > > - Another down side is that the > > JVM > > > > > never > > > > > > > > > trigger > > > > > > > > > > GC > > > > > > > > > > > > due > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > > > > > > > > > > direct memory limit, because the > > > limit > > > > > is > > > > > > > too > > > > > > > > > high > > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > > > > > > > > > > > means we kind of relay on heap > > > memory > > > > to > > > > > > > > trigger > > > > > > > > > > GC > > > > > > > > > > > > and > > > > > > > > > > > > > > > > release > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > > > > > > > > > > memory. That could be a problem > in > > > > cases > > > > > > > where > > > > > > > > > we > > > > > > > > > > > have > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > > > > usage but not enough heap > activity > > > to > > > > > > > trigger > > > > > > > > > the > > > > > > > > > > > GC. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe you can share your reasons > for > > > > > > preferring > > > > > > > > > > > setting a > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > > > > > > > > > > > if there are anything else I > > > overlooked. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Memory Calculation* > > > > > > > > > > > > > > > > > > > > > If there is any conflict between > > > multiple > > > > > > > > > > configuration > > > > > > > > > > > > > that > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > > > > > > explicitly specified, I think we > > should > > > > > throw > > > > > > > an > > > > > > > > > > error. > > > > > > > > > > > > > > > > > > > > > I think doing checking on the > client > > > side > > > > > is > > > > > > a > > > > > > > > good > > > > > > > > > > > idea, > > > > > > > > > > > > > so > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > > > > > > > > > > > K8s we can discover the problem > > before > > > > > > > submitting > > > > > > > > > the > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > is always a good thing. > > > > > > > > > > > > > > > > > > > > > But we can not only rely on the > > client > > > > side > > > > > > > > > checking, > > > > > > > > > > > > > because > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > standalone cluster TaskManagers on > > > > > different > > > > > > > > > machines > > > > > > > > > > > may > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > > > configurations and the client does > > see > > > > > that. > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 PM Yang > > > Wang > > > > < > > > > > > > > > > > > > > > danrtsey...@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi xintong, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your detailed > proposal. > > > > After > > > > > > all > > > > > > > > the > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > introduced, it will be more > > powerful > > > to > > > > > > > control > > > > > > > > > the > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > > > > > > > > > > > > just have few questions about it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Native and Direct Memory > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We do not differentiate user > direct > > > > > memory > > > > > > > and > > > > > > > > > > native > > > > > > > > > > > > > > memory. > > > > > > > > > > > > > > > > > They > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > included in task off-heap memory. > > > > Right? > > > > > > So i > > > > > > > > > don’t > > > > > > > > > > > > think > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > > > > > > > > > > > > the -XX:MaxDirectMemorySize > > > properly. I > > > > > > > prefer > > > > > > > > > > > leaving > > > > > > > > > > > > > it a > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Memory Calculation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the sum of and fine-grained > > > > > > memory(network > > > > > > > > > > memory, > > > > > > > > > > > > > > managed > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > > > > > > > > > > > > is larger than total process > > memory, > > > > how > > > > > do > > > > > > > we > > > > > > > > > deal > > > > > > > > > > > > with > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > > > > > > > > > > > > we need to check the memory > > > > configuration > > > > > > in > > > > > > > > > > client? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song < > > tonysong...@gmail.com> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We would like to start a > > discussion > > > > > > thread > > > > > > > on > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > > > > > > > > > > > > Configuration for > > > TaskExecutors"[1], > > > > > > where > > > > > > > we > > > > > > > > > > > > describe > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > > configurations. > > > > The > > > > > > > FLIP > > > > > > > > > > > document > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > > > early design "Memory Management > > and > > > > > > > > > Configuration > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > > > > > > > > > > > > > with updates from follow-up > > > > discussions > > > > > > > both > > > > > > > > > > online > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This FLIP addresses several > > > > > shortcomings > > > > > > of > > > > > > > > > > current > > > > > > > > > > > > > > (Flink > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor memory > > configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Different configuration > for > > > > > > Streaming > > > > > > > > and > > > > > > > > > > > Batch. > > > > > > > > > > > > > > > > > > > > > > > - Complex and difficult > > > > > configuration > > > > > > of > > > > > > > > > > RocksDB > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > > > > > > > > > > > > > - Complicated, uncertain and > > > hard > > > > to > > > > > > > > > > understand. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Key changes to solve the > problems > > > can > > > > > be > > > > > > > > > > summarized > > > > > > > > > > > > as > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Extend memory manager to > > also > > > > > > account > > > > > > > > for > > > > > > > > > > > memory > > > > > > > > > > > > > > usage > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > > > > > > > > > > > > > - Modify how TaskExecutor > > memory > > > > is > > > > > > > > > > partitioned > > > > > > > > > > > > > > > accounted > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > > > > > > > > > > > > > memory reservations and > pools. > > > > > > > > > > > > > > > > > > > > > > > - Simplify memory > > configuration > > > > > > options > > > > > > > > and > > > > > > > > > > > > > > calculations > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please find more details in the > > > FLIP > > > > > wiki > > > > > > > > > > document > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please note that the early > > design > > > > doc > > > > > > [2] > > > > > > > is > > > > > > > > > out > > > > > > > > > > > of > > > > > > > > > > > > > > sync, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > appreciated to have the > > discussion > > > in > > > > > > this > > > > > > > > > > mailing > > > > > > > > > > > > list > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking forward to your > > feedbacks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
