@Xintong: Concerning "wait for memory users before task dispose and memory
release": I agree, that's how it should be. Let's try it out.

@Xintong @Jingsong: Concerning " JVM does not wait for GC when allocating
direct memory buffer": There seems to be pretty elaborate logic to free
buffers when allocating new ones. See
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643

@Till: Maybe. If we assume that the JVM default works (like going with
option 2 and not setting "-XX:MaxDirectMemorySize" at all), then I think it
should be okay to set "-XX:MaxDirectMemorySize" to
"off_heap_managed_memory + direct_memory" even if we use RocksDB. That is a
big if, though, I honestly have no idea :D Would be good to understand
this, though, because this would affect option (2) and option (1.2).

On Mon, Aug 19, 2019 at 4:44 PM Xintong Song <tonysong...@gmail.com> wrote:

> Thanks for the inputs, Jingsong.
>
> Let me try to summarize your points. Please correct me if I'm wrong.
>
>    - Memory consumers should always avoid returning memory segments to
>    memory manager while there are still un-cleaned structures / threads
> that
>    may use the memory. Otherwise, it would cause serious problems by having
>    multiple consumers trying to use the same memory segment.
>    - JVM does not wait for GC when allocating direct memory buffer.
>    Therefore even we set proper max direct memory size limit, we may still
>    encounter direct memory oom if the GC cleaning memory slower than the
>    direct memory allocation.
>
> Am I understanding this correctly?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Aug 19, 2019 at 4:21 PM JingsongLee <lzljs3620...@aliyun.com
> .invalid>
> wrote:
>
> > Hi stephan:
> >
> > About option 2:
> >
> > if additional threads not cleanly shut down before we can exit the task:
> > In the current case of memory reuse, it has freed up the memory it
> >  uses. If this memory is used by other tasks and asynchronous threads
> >  of exited task may still be writing, there will be concurrent security
> >  problems, and even lead to errors in user computing results.
> >
> > So I think this is a serious and intolerable bug, No matter what the
> >  option is, it should be avoided.
> >
> > About direct memory cleaned by GC:
> > I don't think it is a good idea, I've encountered so many situations
> >  that it's too late for GC to cause DirectMemory OOM. Release and
> >  allocate DirectMemory depend on the type of user job, which is
> >  often beyond our control.
> >
> > Best,
> > Jingsong Lee
> >
> >
> > ------------------------------------------------------------------
> > From:Stephan Ewen <se...@apache.org>
> > Send Time:2019年8月19日(星期一) 15:56
> > To:dev <dev@flink.apache.org>
> > Subject:Re: [DISCUSS] FLIP-49: Unified Memory Configuration for
> > TaskExecutors
> >
> > My main concern with option 2 (manually release memory) is that segfaults
> > in the JVM send off all sorts of alarms on user ends. So we need to
> > guarantee that this never happens.
> >
> > The trickyness is in tasks that uses data structures / algorithms with
> > additional threads, like hash table spill/read and sorting threads. We
> need
> > to ensure that these cleanly shut down before we can exit the task.
> > I am not sure that we have that guaranteed already, that's why option 1.1
> > seemed simpler to me.
> >
> > On Mon, Aug 19, 2019 at 3:42 PM Xintong Song <tonysong...@gmail.com>
> > wrote:
> >
> > > Thanks for the comments, Stephan. Summarized in this way really makes
> > > things easier to understand.
> > >
> > > I'm in favor of option 2, at least for the moment. I think it is not
> that
> > > difficult to keep it segfault safe for memory manager, as long as we
> > always
> > > de-allocate the memory segment when it is released from the memory
> > > consumers. Only if the memory consumer continue using the buffer of
> > memory
> > > segment after releasing it, in which case we do want the job to fail so
> > we
> > > detect the memory leak early.
> > >
> > > For option 1.2, I don't think this is a good idea. Not only because the
> > > assumption (regular GC is enough to clean direct buffers) may not
> always
> > be
> > > true, but also it makes harder for finding problems in cases of memory
> > > overuse. E.g., user configured some direct memory for the user
> libraries.
> > > If the library actually use more direct memory then configured, which
> > > cannot be cleaned by GC because they are still in use, may lead to
> > overuse
> > > of the total container memory. In that case, if it didn't touch the JVM
> > > default max direct memory limit, we cannot get a direct memory OOM and
> it
> > > will become super hard to understand which part of the configuration
> need
> > > to be updated.
> > >
> > > For option 1.1, it has the similar problem as 1.2, if the exceeded
> direct
> > > memory does not reach the max direct memory limit specified by the
> > > dedicated parameter. I think it is slightly better than 1.2, only
> because
> > > we can tune the parameter.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > About the "-XX:MaxDirectMemorySize" discussion, maybe let me
> summarize
> > > it a
> > > > bit differently:
> > > >
> > > > We have the following two options:
> > > >
> > > > (1) We let MemorySegments be de-allocated by the GC. That makes it
> > > segfault
> > > > safe. But then we need a way to trigger GC in case de-allocation and
> > > > re-allocation of a bunch of segments happens quickly, which is often
> > the
> > > > case during batch scheduling or task restart.
> > > >   - The "-XX:MaxDirectMemorySize" (option 1.1) is one way to do this
> > > >   - Another way could be to have a dedicated bookkeeping in the
> > > > MemoryManager (option 1.2), so that this is a number independent of
> the
> > > > "-XX:MaxDirectMemorySize" parameter.
> > > >
> > > > (2) We manually allocate and de-allocate the memory for the
> > > MemorySegments
> > > > (option 2). That way we need not worry about triggering GC by some
> > > > threshold or bookkeeping, but it is harder to prevent segfaults. We
> > need
> > > to
> > > > be very careful about when we release the memory segments (only in
> the
> > > > cleanup phase of the main thread).
> > > >
> > > > If we go with option 1.1, we probably need to set
> > > > "-XX:MaxDirectMemorySize" to "off_heap_managed_memory +
> direct_memory"
> > > and
> > > > have "direct_memory" as a separate reserved memory pool. Because if
> we
> > > just
> > > > set "-XX:MaxDirectMemorySize" to "off_heap_managed_memory +
> > > jvm_overhead",
> > > > then there will be times when that entire memory is allocated by
> direct
> > > > buffers and we have nothing left for the JVM overhead. So we either
> > need
> > > a
> > > > way to compensate for that (again some safety margin cutoff value) or
> > we
> > > > will exceed container memory.
> > > >
> > > > If we go with option 1.2, we need to be aware that it takes elaborate
> > > logic
> > > > to push recycling of direct buffers without always triggering a full
> > GC.
> > > >
> > > >
> > > > My first guess is that the options will be easiest to do in the
> > following
> > > > order:
> > > >
> > > >   - Option 1.1 with a dedicated direct_memory parameter, as discussed
> > > > above. We would need to find a way to set the direct_memory parameter
> > by
> > > > default. We could start with 64 MB and see how it goes in practice.
> One
> > > > danger I see is that setting this loo low can cause a bunch of
> > additional
> > > > GCs compared to before (we need to watch this carefully).
> > > >
> > > >   - Option 2. It is actually quite simple to implement, we could try
> > how
> > > > segfault safe we are at the moment.
> > > >
> > > >   - Option 1.2: We would not touch the "-XX:MaxDirectMemorySize"
> > > parameter
> > > > at all and assume that all the direct memory allocations that the JVM
> > and
> > > > Netty do are infrequent enough to be cleaned up fast enough through
> > > regular
> > > > GC. I am not sure if that is a valid assumption, though.
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song <tonysong...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for sharing your opinion Till.
> > > > >
> > > > > I'm also in favor of alternative 2. I was wondering whether we can
> > > avoid
> > > > > using Unsafe.allocate() for off-heap managed memory and network
> > memory
> > > > with
> > > > > alternative 3. But after giving it a second thought, I think even
> for
> > > > > alternative 3 using direct memory for off-heap managed memory could
> > > cause
> > > > > problems.
> > > > >
> > > > > Hi Yang,
> > > > >
> > > > > Regarding your concern, I think what proposed in this FLIP it to
> have
> > > > both
> > > > > off-heap managed memory and network memory allocated through
> > > > > Unsafe.allocate(), which means they are practically native memory
> and
> > > not
> > > > > limited by JVM max direct memory. The only parts of memory limited
> by
> > > JVM
> > > > > max direct memory are task off-heap memory and JVM overhead, which
> > are
> > > > > exactly alternative 2 suggests to set the JVM max direct memory to.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 16, 2019 at 1:48 PM Till Rohrmann <
> trohrm...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the clarification Xintong. I understand the two
> > > alternatives
> > > > > > now.
> > > > > >
> > > > > > I would be in favour of option 2 because it makes things
> explicit.
> > If
> > > > we
> > > > > > don't limit the direct memory, I fear that we might end up in a
> > > similar
> > > > > > situation as we are currently in: The user might see that her
> > process
> > > > > gets
> > > > > > killed by the OS and does not know why this is the case.
> > > Consequently,
> > > > > she
> > > > > > tries to decrease the process memory size (similar to increasing
> > the
> > > > > cutoff
> > > > > > ratio) in order to accommodate for the extra direct memory. Even
> > > worse,
> > > > > she
> > > > > > tries to decrease memory budgets which are not fully used and
> hence
> > > > won't
> > > > > > change the overall memory consumption.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong Song <
> > tonysong...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Let me explain this with a concrete example Till.
> > > > > > >
> > > > > > > Let's say we have the following scenario.
> > > > > > >
> > > > > > > Total Process Memory: 1GB
> > > > > > > JVM Direct Memory (Task Off-Heap Memory + JVM Overhead): 200MB
> > > > > > > Other Memory (JVM Heap Memory, JVM Metaspace, Off-Heap Managed
> > > Memory
> > > > > and
> > > > > > > Network Memory): 800MB
> > > > > > >
> > > > > > >
> > > > > > > For alternative 2, we set -XX:MaxDirectMemorySize to 200MB.
> > > > > > > For alternative 3, we set -XX:MaxDirectMemorySize to a very
> large
> > > > > value,
> > > > > > > let's say 1TB.
> > > > > > >
> > > > > > > If the actual direct memory usage of Task Off-Heap Memory and
> JVM
> > > > > > Overhead
> > > > > > > do not exceed 200MB, then alternative 2 and alternative 3
> should
> > > have
> > > > > the
> > > > > > > same utility. Setting larger -XX:MaxDirectMemorySize will not
> > > reduce
> > > > > the
> > > > > > > sizes of the other memory pools.
> > > > > > >
> > > > > > > If the actual direct memory usage of Task Off-Heap Memory and
> JVM
> > > > > > > Overhead potentially exceed 200MB, then
> > > > > > >
> > > > > > >    - Alternative 2 suffers from frequent OOM. To avoid that,
> the
> > > only
> > > > > > thing
> > > > > > >    user can do is to modify the configuration and increase JVM
> > > Direct
> > > > > > > Memory
> > > > > > >    (Task Off-Heap Memory + JVM Overhead). Let's say that user
> > > > increases
> > > > > > JVM
> > > > > > >    Direct Memory to 250MB, this will reduce the total size of
> > other
> > > > > > memory
> > > > > > >    pools to 750MB, given the total process memory remains 1GB.
> > > > > > >    - For alternative 3, there is no chance of direct OOM. There
> > are
> > > > > > chances
> > > > > > >    of exceeding the total process memory limit, but given that
> > the
> > > > > > process
> > > > > > > may
> > > > > > >    not use up all the reserved native memory (Off-Heap Managed
> > > > Memory,
> > > > > > > Network
> > > > > > >    Memory, JVM Metaspace), if the actual direct memory usage is
> > > > > slightly
> > > > > > > above
> > > > > > >    yet very close to 200MB, user probably do not need to change
> > the
> > > > > > >    configurations.
> > > > > > >
> > > > > > > Therefore, I think from the user's perspective, a feasible
> > > > > configuration
> > > > > > > for alternative 2 may lead to lower resource utilization
> compared
> > > to
> > > > > > > alternative 3.
> > > > > > >
> > > > > > > Thank you~
> > > > > > >
> > > > > > > Xintong Song
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till Rohrmann <
> > > trohrm...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I guess you have to help me understand the difference between
> > > > > > > alternative 2
> > > > > > > > and 3 wrt to memory under utilization Xintong.
> > > > > > > >
> > > > > > > > - Alternative 2: set XX:MaxDirectMemorySize to Task Off-Heap
> > > Memory
> > > > > and
> > > > > > > JVM
> > > > > > > > Overhead. Then there is the risk that this size is too low
> > > > resulting
> > > > > > in a
> > > > > > > > lot of garbage collection and potentially an OOM.
> > > > > > > > - Alternative 3: set XX:MaxDirectMemorySize to something
> larger
> > > > than
> > > > > > > > alternative 2. This would of course reduce the sizes of the
> > other
> > > > > > memory
> > > > > > > > types.
> > > > > > > >
> > > > > > > > How would alternative 2 now result in an under utilization of
> > > > memory
> > > > > > > > compared to alternative 3? If alternative 3 strictly sets a
> > > higher
> > > > > max
> > > > > > > > direct memory size and we use only little, then I would
> expect
> > > that
> > > > > > > > alternative 3 results in memory under utilization.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang Wang <
> > danrtsey...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi xintong,till
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Native and Direct Memory
> > > > > > > > >
> > > > > > > > > My point is setting a very large max direct memory size
> when
> > we
> > > > do
> > > > > > not
> > > > > > > > > differentiate direct and native memory. If the direct
> > > > > > memory,including
> > > > > > > > user
> > > > > > > > > direct memory and framework direct memory,could be
> calculated
> > > > > > > > > correctly,then
> > > > > > > > > i am in favor of setting direct memory with fixed value.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Memory Calculation
> > > > > > > > >
> > > > > > > > > I agree with xintong. For Yarn and k8s,we need to check the
> > > > memory
> > > > > > > > > configurations in client to avoid submitting successfully
> and
> > > > > failing
> > > > > > > in
> > > > > > > > > the flink master.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Yang
> > > > > > > > >
> > > > > > > > > Xintong Song <tonysong...@gmail.com>于2019年8月13日 周二22:07写道:
> > > > > > > > >
> > > > > > > > > > Thanks for replying, Till.
> > > > > > > > > >
> > > > > > > > > > About MemorySegment, I think you are right that we should
> > not
> > > > > > include
> > > > > > > > > this
> > > > > > > > > > issue in the scope of this FLIP. This FLIP should
> > concentrate
> > > > on
> > > > > > how
> > > > > > > to
> > > > > > > > > > configure memory pools for TaskExecutors, with minimum
> > > > > involvement
> > > > > > on
> > > > > > > > how
> > > > > > > > > > memory consumers use it.
> > > > > > > > > >
> > > > > > > > > > About direct memory, I think alternative 3 may not having
> > the
> > > > > same
> > > > > > > over
> > > > > > > > > > reservation issue that alternative 2 does, but at the
> cost
> > of
> > > > > risk
> > > > > > of
> > > > > > > > > over
> > > > > > > > > > using memory at the container level, which is not good.
> My
> > > > point
> > > > > is
> > > > > > > > that
> > > > > > > > > > both "Task Off-Heap Memory" and "JVM Overhead" are not
> easy
> > > to
> > > > > > > config.
> > > > > > > > > For
> > > > > > > > > > alternative 2, users might configure them higher than
> what
> > > > > actually
> > > > > > > > > needed,
> > > > > > > > > > just to avoid getting a direct OOM. For alternative 3,
> > users
> > > do
> > > > > not
> > > > > > > get
> > > > > > > > > > direct OOM, so they may not config the two options
> > > aggressively
> > > > > > high.
> > > > > > > > But
> > > > > > > > > > the consequences are risks of overall container memory
> > usage
> > > > > > exceeds
> > > > > > > > the
> > > > > > > > > > budget.
> > > > > > > > > >
> > > > > > > > > > Thank you~
> > > > > > > > > >
> > > > > > > > > > Xintong Song
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM Till Rohrmann <
> > > > > > trohrm...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for proposing this FLIP Xintong.
> > > > > > > > > > >
> > > > > > > > > > > All in all I think it already looks quite good.
> > Concerning
> > > > the
> > > > > > > first
> > > > > > > > > open
> > > > > > > > > > > question about allocating memory segments, I was
> > wondering
> > > > > > whether
> > > > > > > > this
> > > > > > > > > > is
> > > > > > > > > > > strictly necessary to do in the context of this FLIP or
> > > > whether
> > > > > > > this
> > > > > > > > > > could
> > > > > > > > > > > be done as a follow up? Without knowing all details, I
> > > would
> > > > be
> > > > > > > > > concerned
> > > > > > > > > > > that we would widen the scope of this FLIP too much
> > because
> > > > we
> > > > > > > would
> > > > > > > > > have
> > > > > > > > > > > to touch all the existing call sites of the
> MemoryManager
> > > > where
> > > > > > we
> > > > > > > > > > allocate
> > > > > > > > > > > memory segments (this should mainly be batch
> operators).
> > > The
> > > > > > > addition
> > > > > > > > > of
> > > > > > > > > > > the memory reservation call to the MemoryManager should
> > not
> > > > be
> > > > > > > > affected
> > > > > > > > > > by
> > > > > > > > > > > this and I would hope that this is the only point of
> > > > > interaction
> > > > > > a
> > > > > > > > > > > streaming job would have with the MemoryManager.
> > > > > > > > > > >
> > > > > > > > > > > Concerning the second open question about setting or
> not
> > > > > setting
> > > > > > a
> > > > > > > > max
> > > > > > > > > > > direct memory limit, I would also be interested why
> Yang
> > > Wang
> > > > > > > thinks
> > > > > > > > > > > leaving it open would be best. My concern about this
> > would
> > > be
> > > > > > that
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > > be in a similar situation as we are now with the
> > > > > > > RocksDBStateBackend.
> > > > > > > > > If
> > > > > > > > > > > the different memory pools are not clearly separated
> and
> > > can
> > > > > > spill
> > > > > > > > over
> > > > > > > > > > to
> > > > > > > > > > > a different pool, then it is quite hard to understand
> > what
> > > > > > exactly
> > > > > > > > > > causes a
> > > > > > > > > > > process to get killed for using too much memory. This
> > could
> > > > > then
> > > > > > > > easily
> > > > > > > > > > > lead to a similar situation what we have with the
> > > > cutoff-ratio.
> > > > > > So
> > > > > > > > why
> > > > > > > > > > not
> > > > > > > > > > > setting a sane default value for max direct memory and
> > > giving
> > > > > the
> > > > > > > > user
> > > > > > > > > an
> > > > > > > > > > > option to increase it if he runs into an OOM.
> > > > > > > > > > >
> > > > > > > > > > > @Xintong, how would alternative 2 lead to lower memory
> > > > > > utilization
> > > > > > > > than
> > > > > > > > > > > alternative 3 where we set the direct memory to a
> higher
> > > > value?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Till
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM Xintong Song <
> > > > > > tonysong...@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for the feedback, Yang.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding your comments:
> > > > > > > > > > > >
> > > > > > > > > > > > *Native and Direct Memory*
> > > > > > > > > > > > I think setting a very large max direct memory size
> > > > > definitely
> > > > > > > has
> > > > > > > > > some
> > > > > > > > > > > > good sides. E.g., we do not worry about direct OOM,
> and
> > > we
> > > > > > don't
> > > > > > > > even
> > > > > > > > > > > need
> > > > > > > > > > > > to allocate managed / network memory with
> > > > Unsafe.allocate() .
> > > > > > > > > > > > However, there are also some down sides of doing
> this.
> > > > > > > > > > > >
> > > > > > > > > > > >    - One thing I can think of is that if a task
> > executor
> > > > > > > container
> > > > > > > > is
> > > > > > > > > > > >    killed due to overusing memory, it could be hard
> for
> > > use
> > > > > to
> > > > > > > know
> > > > > > > > > > which
> > > > > > > > > > > > part
> > > > > > > > > > > >    of the memory is overused.
> > > > > > > > > > > >    - Another down side is that the JVM never trigger
> GC
> > > due
> > > > > to
> > > > > > > > > reaching
> > > > > > > > > > > max
> > > > > > > > > > > >    direct memory limit, because the limit is too high
> > to
> > > be
> > > > > > > > reached.
> > > > > > > > > > That
> > > > > > > > > > > >    means we kind of relay on heap memory to trigger
> GC
> > > and
> > > > > > > release
> > > > > > > > > > direct
> > > > > > > > > > > >    memory. That could be a problem in cases where we
> > have
> > > > > more
> > > > > > > > direct
> > > > > > > > > > > > memory
> > > > > > > > > > > >    usage but not enough heap activity to trigger the
> > GC.
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe you can share your reasons for preferring
> > setting a
> > > > > very
> > > > > > > > large
> > > > > > > > > > > value,
> > > > > > > > > > > > if there are anything else I overlooked.
> > > > > > > > > > > >
> > > > > > > > > > > > *Memory Calculation*
> > > > > > > > > > > > If there is any conflict between multiple
> configuration
> > > > that
> > > > > > user
> > > > > > > > > > > > explicitly specified, I think we should throw an
> error.
> > > > > > > > > > > > I think doing checking on the client side is a good
> > idea,
> > > > so
> > > > > > that
> > > > > > > > on
> > > > > > > > > > > Yarn /
> > > > > > > > > > > > K8s we can discover the problem before submitting the
> > > Flink
> > > > > > > > cluster,
> > > > > > > > > > > which
> > > > > > > > > > > > is always a good thing.
> > > > > > > > > > > > But we can not only rely on the client side checking,
> > > > because
> > > > > > for
> > > > > > > > > > > > standalone cluster TaskManagers on different machines
> > may
> > > > > have
> > > > > > > > > > different
> > > > > > > > > > > > configurations and the client does see that.
> > > > > > > > > > > > What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you~
> > > > > > > > > > > >
> > > > > > > > > > > > Xintong Song
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 PM Yang Wang <
> > > > > > danrtsey...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi xintong,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your detailed proposal. After all the
> > memory
> > > > > > > > > configuration
> > > > > > > > > > > are
> > > > > > > > > > > > > introduced, it will be more powerful to control the
> > > flink
> > > > > > > memory
> > > > > > > > > > > usage. I
> > > > > > > > > > > > > just have few questions about it.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >    - Native and Direct Memory
> > > > > > > > > > > > >
> > > > > > > > > > > > > We do not differentiate user direct memory and
> native
> > > > > memory.
> > > > > > > > They
> > > > > > > > > > are
> > > > > > > > > > > > all
> > > > > > > > > > > > > included in task off-heap memory. Right? So i don’t
> > > think
> > > > > we
> > > > > > > > could
> > > > > > > > > > not
> > > > > > > > > > > > set
> > > > > > > > > > > > > the -XX:MaxDirectMemorySize properly. I prefer
> > leaving
> > > > it a
> > > > > > > very
> > > > > > > > > > large
> > > > > > > > > > > > > value.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >    - Memory Calculation
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the sum of and fine-grained memory(network
> memory,
> > > > > managed
> > > > > > > > > memory,
> > > > > > > > > > > > etc.)
> > > > > > > > > > > > > is larger than total process memory, how do we deal
> > > with
> > > > > this
> > > > > > > > > > > situation?
> > > > > > > > > > > > Do
> > > > > > > > > > > > > we need to check the memory configuration in
> client?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com> 于2019年8月7日周三
> > > > > 下午10:14写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We would like to start a discussion thread on
> > > "FLIP-49:
> > > > > > > Unified
> > > > > > > > > > > Memory
> > > > > > > > > > > > > > Configuration for TaskExecutors"[1], where we
> > > describe
> > > > > how
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > > > > > TaskExecutor memory configurations. The FLIP
> > document
> > > > is
> > > > > > > mostly
> > > > > > > > > > based
> > > > > > > > > > > > on
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > early design "Memory Management and Configuration
> > > > > > > Reloaded"[2]
> > > > > > > > by
> > > > > > > > > > > > > Stephan,
> > > > > > > > > > > > > > with updates from follow-up discussions both
> online
> > > and
> > > > > > > > offline.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This FLIP addresses several shortcomings of
> current
> > > > > (Flink
> > > > > > > 1.9)
> > > > > > > > > > > > > > TaskExecutor memory configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >    - Different configuration for Streaming and
> > Batch.
> > > > > > > > > > > > > >    - Complex and difficult configuration of
> RocksDB
> > > in
> > > > > > > > Streaming.
> > > > > > > > > > > > > >    - Complicated, uncertain and hard to
> understand.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Key changes to solve the problems can be
> summarized
> > > as
> > > > > > > follows.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >    - Extend memory manager to also account for
> > memory
> > > > > usage
> > > > > > > by
> > > > > > > > > > state
> > > > > > > > > > > > > >    backends.
> > > > > > > > > > > > > >    - Modify how TaskExecutor memory is
> partitioned
> > > > > > accounted
> > > > > > > > > > > individual
> > > > > > > > > > > > > >    memory reservations and pools.
> > > > > > > > > > > > > >    - Simplify memory configuration options and
> > > > > calculations
> > > > > > > > > logics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please find more details in the FLIP wiki
> document
> > > [1].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (Please note that the early design doc [2] is out
> > of
> > > > > sync,
> > > > > > > and
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > > > appreciated to have the discussion in this
> mailing
> > > list
> > > > > > > > thread.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looking forward to your feedbacks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you~
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Xintong Song
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [2]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song <tonysong...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for sharing your opinion Till.
> > > > >
> > > > > I'm also in favor of alternative 2. I was wondering whether we can
> > > avoid
> > > > > using Unsafe.allocate() for off-heap managed memory and network
> > memory
> > > > with
> > > > > alternative 3. But after giving it a second thought, I think even
> for
> > > > > alternative 3 using direct memory for off-heap managed memory could
> > > cause
> > > > > problems.
> > > > >
> > > > > Hi Yang,
> > > > >
> > > > > Regarding your concern, I think what proposed in this FLIP it to
> have
> > > > both
> > > > > off-heap managed memory and network memory allocated through
> > > > > Unsafe.allocate(), which means they are practically native memory
> and
> > > not
> > > > > limited by JVM max direct memory. The only parts of memory limited
> by
> > > JVM
> > > > > max direct memory are task off-heap memory and JVM overhead, which
> > are
> > > > > exactly alternative 2 suggests to set the JVM max direct memory to.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 16, 2019 at 1:48 PM Till Rohrmann <
> trohrm...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the clarification Xintong. I understand the two
> > > alternatives
> > > > > > now.
> > > > > >
> > > > > > I would be in favour of option 2 because it makes things
> explicit.
> > If
> > > > we
> > > > > > don't limit the direct memory, I fear that we might end up in a
> > > similar
> > > > > > situation as we are currently in: The user might see that her
> > process
> > > > > gets
> > > > > > killed by the OS and does not know why this is the case.
> > > Consequently,
> > > > > she
> > > > > > tries to decrease the process memory size (similar to increasing
> > the
> > > > > cutoff
> > > > > > ratio) in order to accommodate for the extra direct memory. Even
> > > worse,
> > > > > she
> > > > > > tries to decrease memory budgets which are not fully used and
> hence
> > > > won't
> > > > > > change the overall memory consumption.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Fri, Aug 16, 2019 at 11:01 AM Xintong Song <
> > tonysong...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Let me explain this with a concrete example Till.
> > > > > > >
> > > > > > > Let's say we have the following scenario.
> > > > > > >
> > > > > > > Total Process Memory: 1GB
> > > > > > > JVM Direct Memory (Task Off-Heap Memory + JVM Overhead): 200MB
> > > > > > > Other Memory (JVM Heap Memory, JVM Metaspace, Off-Heap Managed
> > > Memory
> > > > > and
> > > > > > > Network Memory): 800MB
> > > > > > >
> > > > > > >
> > > > > > > For alternative 2, we set -XX:MaxDirectMemorySize to 200MB.
> > > > > > > For alternative 3, we set -XX:MaxDirectMemorySize to a very
> large
> > > > > value,
> > > > > > > let's say 1TB.
> > > > > > >
> > > > > > > If the actual direct memory usage of Task Off-Heap Memory and
> JVM
> > > > > > Overhead
> > > > > > > do not exceed 200MB, then alternative 2 and alternative 3
> should
> > > have
> > > > > the
> > > > > > > same utility. Setting larger -XX:MaxDirectMemorySize will not
> > > reduce
> > > > > the
> > > > > > > sizes of the other memory pools.
> > > > > > >
> > > > > > > If the actual direct memory usage of Task Off-Heap Memory and
> JVM
> > > > > > > Overhead potentially exceed 200MB, then
> > > > > > >
> > > > > > >    - Alternative 2 suffers from frequent OOM. To avoid that,
> the
> > > only
> > > > > > thing
> > > > > > >    user can do is to modify the configuration and increase JVM
> > > Direct
> > > > > > > Memory
> > > > > > >    (Task Off-Heap Memory + JVM Overhead). Let's say that user
> > > > increases
> > > > > > JVM
> > > > > > >    Direct Memory to 250MB, this will reduce the total size of
> > other
> > > > > > memory
> > > > > > >    pools to 750MB, given the total process memory remains 1GB.
> > > > > > >    - For alternative 3, there is no chance of direct OOM. There
> > are
> > > > > > chances
> > > > > > >    of exceeding the total process memory limit, but given that
> > the
> > > > > > process
> > > > > > > may
> > > > > > >    not use up all the reserved native memory (Off-Heap Managed
> > > > Memory,
> > > > > > > Network
> > > > > > >    Memory, JVM Metaspace), if the actual direct memory usage is
> > > > > slightly
> > > > > > > above
> > > > > > >    yet very close to 200MB, user probably do not need to change
> > the
> > > > > > >    configurations.
> > > > > > >
> > > > > > > Therefore, I think from the user's perspective, a feasible
> > > > > configuration
> > > > > > > for alternative 2 may lead to lower resource utilization
> compared
> > > to
> > > > > > > alternative 3.
> > > > > > >
> > > > > > > Thank you~
> > > > > > >
> > > > > > > Xintong Song
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 16, 2019 at 10:28 AM Till Rohrmann <
> > > trohrm...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I guess you have to help me understand the difference between
> > > > > > > alternative 2
> > > > > > > > and 3 wrt to memory under utilization Xintong.
> > > > > > > >
> > > > > > > > - Alternative 2: set XX:MaxDirectMemorySize to Task Off-Heap
> > > Memory
> > > > > and
> > > > > > > JVM
> > > > > > > > Overhead. Then there is the risk that this size is too low
> > > > resulting
> > > > > > in a
> > > > > > > > lot of garbage collection and potentially an OOM.
> > > > > > > > - Alternative 3: set XX:MaxDirectMemorySize to something
> larger
> > > > than
> > > > > > > > alternative 2. This would of course reduce the sizes of the
> > other
> > > > > > memory
> > > > > > > > types.
> > > > > > > >
> > > > > > > > How would alternative 2 now result in an under utilization of
> > > > memory
> > > > > > > > compared to alternative 3? If alternative 3 strictly sets a
> > > higher
> > > > > max
> > > > > > > > direct memory size and we use only little, then I would
> expect
> > > that
> > > > > > > > alternative 3 results in memory under utilization.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Tue, Aug 13, 2019 at 4:19 PM Yang Wang <
> > danrtsey...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi xintong,till
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Native and Direct Memory
> > > > > > > > >
> > > > > > > > > My point is setting a very large max direct memory size
> when
> > we
> > > > do
> > > > > > not
> > > > > > > > > differentiate direct and native memory. If the direct
> > > > > > memory,including
> > > > > > > > user
> > > > > > > > > direct memory and framework direct memory,could be
> calculated
> > > > > > > > > correctly,then
> > > > > > > > > i am in favor of setting direct memory with fixed value.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Memory Calculation
> > > > > > > > >
> > > > > > > > > I agree with xintong. For Yarn and k8s,we need to check the
> > > > memory
> > > > > > > > > configurations in client to avoid submitting successfully
> and
> > > > > failing
> > > > > > > in
> > > > > > > > > the flink master.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Yang
> > > > > > > > >
> > > > > > > > > Xintong Song <tonysong...@gmail.com>于2019年8月13日 周二22:07写道:
> > > > > > > > >
> > > > > > > > > > Thanks for replying, Till.
> > > > > > > > > >
> > > > > > > > > > About MemorySegment, I think you are right that we should
> > not
> > > > > > include
> > > > > > > > > this
> > > > > > > > > > issue in the scope of this FLIP. This FLIP should
> > concentrate
> > > > on
> > > > > > how
> > > > > > > to
> > > > > > > > > > configure memory pools for TaskExecutors, with minimum
> > > > > involvement
> > > > > > on
> > > > > > > > how
> > > > > > > > > > memory consumers use it.
> > > > > > > > > >
> > > > > > > > > > About direct memory, I think alternative 3 may not having
> > the
> > > > > same
> > > > > > > over
> > > > > > > > > > reservation issue that alternative 2 does, but at the
> cost
> > of
> > > > > risk
> > > > > > of
> > > > > > > > > over
> > > > > > > > > > using memory at the container level, which is not good.
> My
> > > > point
> > > > > is
> > > > > > > > that
> > > > > > > > > > both "Task Off-Heap Memory" and "JVM Overhead" are not
> easy
> > > to
> > > > > > > config.
> > > > > > > > > For
> > > > > > > > > > alternative 2, users might configure them higher than
> what
> > > > > actually
> > > > > > > > > needed,
> > > > > > > > > > just to avoid getting a direct OOM. For alternative 3,
> > users
> > > do
> > > > > not
> > > > > > > get
> > > > > > > > > > direct OOM, so they may not config the two options
> > > aggressively
> > > > > > high.
> > > > > > > > But
> > > > > > > > > > the consequences are risks of overall container memory
> > usage
> > > > > > exceeds
> > > > > > > > the
> > > > > > > > > > budget.
> > > > > > > > > >
> > > > > > > > > > Thank you~
> > > > > > > > > >
> > > > > > > > > > Xintong Song
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 13, 2019 at 9:39 AM Till Rohrmann <
> > > > > > trohrm...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for proposing this FLIP Xintong.
> > > > > > > > > > >
> > > > > > > > > > > All in all I think it already looks quite good.
> > Concerning
> > > > the
> > > > > > > first
> > > > > > > > > open
> > > > > > > > > > > question about allocating memory segments, I was
> > wondering
> > > > > > whether
> > > > > > > > this
> > > > > > > > > > is
> > > > > > > > > > > strictly necessary to do in the context of this FLIP or
> > > > whether
> > > > > > > this
> > > > > > > > > > could
> > > > > > > > > > > be done as a follow up? Without knowing all details, I
> > > would
> > > > be
> > > > > > > > > concerned
> > > > > > > > > > > that we would widen the scope of this FLIP too much
> > because
> > > > we
> > > > > > > would
> > > > > > > > > have
> > > > > > > > > > > to touch all the existing call sites of the
> MemoryManager
> > > > where
> > > > > > we
> > > > > > > > > > allocate
> > > > > > > > > > > memory segments (this should mainly be batch
> operators).
> > > The
> > > > > > > addition
> > > > > > > > > of
> > > > > > > > > > > the memory reservation call to the MemoryManager should
> > not
> > > > be
> > > > > > > > affected
> > > > > > > > > > by
> > > > > > > > > > > this and I would hope that this is the only point of
> > > > > interaction
> > > > > > a
> > > > > > > > > > > streaming job would have with the MemoryManager.
> > > > > > > > > > >
> > > > > > > > > > > Concerning the second open question about setting or
> not
> > > > > setting
> > > > > > a
> > > > > > > > max
> > > > > > > > > > > direct memory limit, I would also be interested why
> Yang
> > > Wang
> > > > > > > thinks
> > > > > > > > > > > leaving it open would be best. My concern about this
> > would
> > > be
> > > > > > that
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > > be in a similar situation as we are now with the
> > > > > > > RocksDBStateBackend.
> > > > > > > > > If
> > > > > > > > > > > the different memory pools are not clearly separated
> and
> > > can
> > > > > > spill
> > > > > > > > over
> > > > > > > > > > to
> > > > > > > > > > > a different pool, then it is quite hard to understand
> > what
> > > > > > exactly
> > > > > > > > > > causes a
> > > > > > > > > > > process to get killed for using too much memory. This
> > could
> > > > > then
> > > > > > > > easily
> > > > > > > > > > > lead to a similar situation what we have with the
> > > > cutoff-ratio.
> > > > > > So
> > > > > > > > why
> > > > > > > > > > not
> > > > > > > > > > > setting a sane default value for max direct memory and
> > > giving
> > > > > the
> > > > > > > > user
> > > > > > > > > an
> > > > > > > > > > > option to increase it if he runs into an OOM.
> > > > > > > > > > >
> > > > > > > > > > > @Xintong, how would alternative 2 lead to lower memory
> > > > > > utilization
> > > > > > > > than
> > > > > > > > > > > alternative 3 where we set the direct memory to a
> higher
> > > > value?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Till
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 9, 2019 at 9:12 AM Xintong Song <
> > > > > > tonysong...@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for the feedback, Yang.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding your comments:
> > > > > > > > > > > >
> > > > > > > > > > > > *Native and Direct Memory*
> > > > > > > > > > > > I think setting a very large max direct memory size
> > > > > definitely
> > > > > > > has
> > > > > > > > > some
> > > > > > > > > > > > good sides. E.g., we do not worry about direct OOM,
> and
> > > we
> > > > > > don't
> > > > > > > > even
> > > > > > > > > > > need
> > > > > > > > > > > > to allocate managed / network memory with
> > > > Unsafe.allocate() .
> > > > > > > > > > > > However, there are also some down sides of doing
> this.
> > > > > > > > > > > >
> > > > > > > > > > > >    - One thing I can think of is that if a task
> > executor
> > > > > > > container
> > > > > > > > is
> > > > > > > > > > > >    killed due to overusing memory, it could be hard
> for
> > > use
> > > > > to
> > > > > > > know
> > > > > > > > > > which
> > > > > > > > > > > > part
> > > > > > > > > > > >    of the memory is overused.
> > > > > > > > > > > >    - Another down side is that the JVM never trigger
> GC
> > > due
> > > > > to
> > > > > > > > > reaching
> > > > > > > > > > > max
> > > > > > > > > > > >    direct memory limit, because the limit is too high
> > to
> > > be
> > > > > > > > reached.
> > > > > > > > > > That
> > > > > > > > > > > >    means we kind of relay on heap memory to trigger
> GC
> > > and
> > > > > > > release
> > > > > > > > > > direct
> > > > > > > > > > > >    memory. That could be a problem in cases where we
> > have
> > > > > more
> > > > > > > > direct
> > > > > > > > > > > > memory
> > > > > > > > > > > >    usage but not enough heap activity to trigger the
> > GC.
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe you can share your reasons for preferring
> > setting a
> > > > > very
> > > > > > > > large
> > > > > > > > > > > value,
> > > > > > > > > > > > if there are anything else I overlooked.
> > > > > > > > > > > >
> > > > > > > > > > > > *Memory Calculation*
> > > > > > > > > > > > If there is any conflict between multiple
> configuration
> > > > that
> > > > > > user
> > > > > > > > > > > > explicitly specified, I think we should throw an
> error.
> > > > > > > > > > > > I think doing checking on the client side is a good
> > idea,
> > > > so
> > > > > > that
> > > > > > > > on
> > > > > > > > > > > Yarn /
> > > > > > > > > > > > K8s we can discover the problem before submitting the
> > > Flink
> > > > > > > > cluster,
> > > > > > > > > > > which
> > > > > > > > > > > > is always a good thing.
> > > > > > > > > > > > But we can not only rely on the client side checking,
> > > > because
> > > > > > for
> > > > > > > > > > > > standalone cluster TaskManagers on different machines
> > may
> > > > > have
> > > > > > > > > > different
> > > > > > > > > > > > configurations and the client does see that.
> > > > > > > > > > > > What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you~
> > > > > > > > > > > >
> > > > > > > > > > > > Xintong Song
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 8, 2019 at 5:09 PM Yang Wang <
> > > > > > danrtsey...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi xintong,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your detailed proposal. After all the
> > memory
> > > > > > > > > configuration
> > > > > > > > > > > are
> > > > > > > > > > > > > introduced, it will be more powerful to control the
> > > flink
> > > > > > > memory
> > > > > > > > > > > usage. I
> > > > > > > > > > > > > just have few questions about it.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >    - Native and Direct Memory
> > > > > > > > > > > > >
> > > > > > > > > > > > > We do not differentiate user direct memory and
> native
> > > > > memory.
> > > > > > > > They
> > > > > > > > > > are
> > > > > > > > > > > > all
> > > > > > > > > > > > > included in task off-heap memory. Right? So i don’t
> > > think
> > > > > we
> > > > > > > > could
> > > > > > > > > > not
> > > > > > > > > > > > set
> > > > > > > > > > > > > the -XX:MaxDirectMemorySize properly. I prefer
> > leaving
> > > > it a
> > > > > > > very
> > > > > > > > > > large
> > > > > > > > > > > > > value.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >    - Memory Calculation
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the sum of and fine-grained memory(network
> memory,
> > > > > managed
> > > > > > > > > memory,
> > > > > > > > > > > > etc.)
> > > > > > > > > > > > > is larger than total process memory, how do we deal
> > > with
> > > > > this
> > > > > > > > > > > situation?
> > > > > > > > > > > > Do
> > > > > > > > > > > > > we need to check the memory configuration in
> client?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com> 于2019年8月7日周三
> > > > > 下午10:14写道:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We would like to start a discussion thread on
> > > "FLIP-49:
> > > > > > > Unified
> > > > > > > > > > > Memory
> > > > > > > > > > > > > > Configuration for TaskExecutors"[1], where we
> > > describe
> > > > > how
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > > > > > TaskExecutor memory configurations. The FLIP
> > document
> > > > is
> > > > > > > mostly
> > > > > > > > > > based
> > > > > > > > > > > > on
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > early design "Memory Management and Configuration
> > > > > > > Reloaded"[2]
> > > > > > > > by
> > > > > > > > > > > > > Stephan,
> > > > > > > > > > > > > > with updates from follow-up discussions both
> online
> > > and
> > > > > > > > offline.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This FLIP addresses several shortcomings of
> current
> > > > > (Flink
> > > > > > > 1.9)
> > > > > > > > > > > > > > TaskExecutor memory configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >    - Different configuration for Streaming and
> > Batch.
> > > > > > > > > > > > > >    - Complex and difficult configuration of
> RocksDB
> > > in
> > > > > > > > Streaming.
> > > > > > > > > > > > > >    - Complicated, uncertain and hard to
> understand.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Key changes to solve the problems can be
> summarized
> > > as
> > > > > > > follows.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >    - Extend memory manager to also account for
> > memory
> > > > > usage
> > > > > > > by
> > > > > > > > > > state
> > > > > > > > > > > > > >    backends.
> > > > > > > > > > > > > >    - Modify how TaskExecutor memory is
> partitioned
> > > > > > accounted
> > > > > > > > > > > individual
> > > > > > > > > > > > > >    memory reservations and pools.
> > > > > > > > > > > > > >    - Simplify memory configuration options and
> > > > > calculations
> > > > > > > > > logics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please find more details in the FLIP wiki
> document
> > > [1].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (Please note that the early design doc [2] is out
> > of
> > > > > sync,
> > > > > > > and
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > > > appreciated to have the discussion in this
> mailing
> > > list
> > > > > > > > thread.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looking forward to your feedbacks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you~
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Xintong Song
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [2]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Reply via email to