fle component
> > (as
> > >> > >> > @Zhijiang
> > >> > >> > > > also
> > >> > >> > > > > suggested), but that is not accounted by this memory
> pool.
> > >> This
> > >> > is
> > >> > >
ell as WebUI to present the memory
> >> pool
> >> > >> > sizes.
> >> > >> > > I
> >> > >> > > > > would suggest to address these as follow-ups of all the
> three
> >> > >> > resource
> >> > &g
t;
>> > >> > > > > @Till
>> > >> > > > > - My understanding is that Task Off-heap memory accounts for
>> > both
>> > >> > > direct
>> > >> > > > > and native memory used by the user code. I&#x
y it out the way we set
> > >> > > '-XX:MaxDirectMemorySize'
> > >> > > > > in current design and may switch to other alternatives if it
> > >> doesn't
> > >> > > work
> > >> > > > > out well, I w
t; add
> >> it
> >> > if
> >> > > > it
> >> > > > > doesn't work well.
> >> > > > > - Agree that it's really important to have good documentation
> for
> >> > this.
> >> > > >
rtant to have good documentation for
>> > this.
>> > > > See
>> > > > > above.
>> > > > >
>> > > > > @Zhijiang
>> > > > > - Thanks for the input. My understanding is that 'shuffle memory'
>> is
>> > a
>> > > > > portion of the task executor memory reserved
; > > > > task executor (outside of the shuffle implementation) should only
> > know
> > > > the
> > > > > overall memory usage of the shuffle component but no need to
> > understand
> > > > > more details inside the shu
tside of the shuffle implementation) should only
> know
> > > the
> > > > overall memory usage of the shuffle component but no need to
> understand
> > > > more details inside the shuffle implementation.
> > > >
> > > > Thank you~
> &g
M zhijiang > > .invalid>
> > > wrote:
> > >
> > > > Thanks for proposing this FLIP and also +1 on my side.
> > > >
> > > > @Andrey Zagrebin For the point of "network memory is actually used
> more
> > > > than shuffling", I guess that the component of queryable state is
> als
10:41 PM zhijiang > > .invalid>
> > > wrote:
> > >
> > > > Thanks for proposing this FLIP and also +1 on my side.
> > > >
> > > > @Andrey Zagrebin For the point of "network memory is actually used
> more
> > > > than shuf
y stack atm, which is outside of shuffling.
> > > In addition, if we only consider the shuffle memory provided by shuffle
> > > service interface, we should not only consider the memory used by local
> > > buffer pool, but also consider the netty internal memory
> > >
ave not the zero-copy improvement
> > on dowstream read side. This issue might be out of the vote scope, just
> > think of we have this issue in [1]. :)
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-12110
> >
> > Best,
> > Zhijiang
> > --
>
> Best,
> Zhijiang
> --------------
> From:Till Rohrmann
> Send Time:2019年9月3日(星期二) 15:07
> To:dev
> Subject:Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors
>
> Thanks for creating this FLIP and s
out of the vote scope, just think of
we have this issue in [1]. :)
[1] https://issues.apache.org/jira/browse/FLINK-12110
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2019年9月3日(星期二) 15:07
To:dev
Subject:Re: [VOTE] FLIP-49: Unif
Thanks for creating this FLIP and starting the vote Xintong.
+1 for the proposal from my side.
I agree with Stephan that we might wanna revisit some of the configuration
names.
If I understood it correctly, then Task Off-heap memory represents the
direct memory used by the user code, right? How
Thanks for starting the vote Xintong
Also +1 for the proposed FLIP-49.
@Stephan regarding namings: network vs shuffle.
My understanding so far was that the network memory is what we basically
give to Shuffle implementations and default netty implementation uses it in
particular mostly for network
+1 to the proposal in general
A few things seems to be a bit put of sync with the latest discussions
though.
The section about JVM Parameters states that the
-XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle
Memory and JVM Overhead.
The way I understand the last discussion con
Hi everyone,
I'm here to re-start the voting process for FLIP-49 [1], with respect to
consensus reached in this thread [2] regarding some new comments and
concerns.
This voting will be open for at least 72 hours. I'll try to close it Sep.
5, 14:00 UTC, unless there is an objection or not enough v
Alright, then let's keep the discussion in the DISCUSS mailing thread, and
see whether we need to restart the vote.
Thank you~
Xintong Song
On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann wrote:
> I had a couple of comments concerning the implementation plan. I've posted
> them to the original
I had a couple of comments concerning the implementation plan. I've posted
them to the original discussion thread. Depending on the outcome of this
discussion we might need to restart the vote.
Cheers,
Till
On Tue, Aug 27, 2019 at 11:14 AM Xintong Song wrote:
> Hi all,
>
> I would like to start
Hi all,
I would like to start the voting process for FLIP-49 [1], which is
discussed and reached consensus in this thread [2].
This voting will be open for at least 72 hours. I'll try to close it Aug.
30 10:00 UTC, unless there is an objection or not enough votes.
Thank you~
Xintong Song
[1]
21 matches
Mail list logo
