+1 The static memory assignment of the network buffer tool caused some problems for users in the past. Ultimately, dynamic memory management would be desirable. Until then, let's remove the absolute value configuration for the network buffers and introduce a parameter to divide the heap memory among network and non-network code.
On Tue, Feb 3, 2015 at 10:48 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Yes, I would really like to get rid of the distinction between operator and > network buffers. > Having all buffers been taken from the same pool is a good step towards > that goal. Until the assignment is dynamic, I prefer to have a config > option for the network / operator ratio. > > +1 for the proposal and a ratio config option. > > 2015-02-03 9:52 GMT+01:00 Stephan Ewen <se...@apache.org>: > >> I like this approach and would suggest to make the ratio configurable. The >> default could be 50/50 or 60/40 (op heap / net heap) >> >> On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u.cel...@fu-berlin.de> wrote: >> >> > Currently, the memory configuration of a task manager encompasses two >> > things: >> > >> > 1) NETWORK buffers: Fixed amount of memory for the network buffer pool >> > (default: 2048 buffers, each 32 KB => 64 MB) >> > >> > 2) OPERATOR buffers: A configurable fraction of the available heap memory >> > (default 0.7) for the memory manager used by internal operators like sort >> > or hash >> > >> > With the recently added supported for intermediate results, intermediate >> > results live in the network stack and use buffers from the network buffer >> > pool. Currently, our memory management is not a problem, because we only >> > support ephemeral intermediate results, which are directly consumed in a >> > pipelined fashion (i.e. the buffer pools are short-lived). >> > >> > But with the upcoming support for persistent intermediate results ( >> > https://github.com/apache/flink/pull/356) and fine-grained fault >> > tolerance, we need to rethink how we configure/divide the available >> memory >> > between the network stack and the operators as the network buffer pools >> > will live longer. >> > >> > I would suggest the following: >> > >> > 1) Remove the configuration for network buffers >> > >> > 2) Keep the fraction configuration, but internally divide it between >> > network stack and operators in a 50:50 fashion (in the future with >> dynamic >> > memory management, we will not even need to statically divide the memory >> > between network stack and operators). >> > >> > >> > What do you think? Is this reasonable? Should the division be >> configurable >> > as well? >> > >> > – Ufuk >>
