Hi Ufuk,
Thanks for this. Really appreciated.
Cheers
On Tue, Feb 23, 2016 at 8:04 PM, Ufuk Celebi wrote:
> I would go with one task manager with 48 slots per machine. This
> reduces the communication overheads between task managers.
>
> Regarding memory configuration: Given that the machines h
I would go with one task manager with 48 slots per machine. This
reduces the communication overheads between task managers.
Regarding memory configuration: Given that the machines have plenty of
memory, I would configure a bigger heap than the 4 GB you had
previously. Furhermore, you can also cons
Hi Ufuk and Fabian,
Is that better to start 48 task manager ( one slot each ) in one machine
than having single task manager with 48 slot ? Any trade-off that we should
know etc ?
Cheers
On Tue, Feb 23, 2016 at 3:03 PM, Welly Tambunan wrote:
> Hi Ufuk,
>
> Thanks for the explanation.
>
> Yes.
Hi Ufuk,
Thanks for the explanation.
Yes. Our jobs is all streaming job.
Cheers
On Tue, Feb 23, 2016 at 2:48 PM, Ufuk Celebi wrote:
> The new default is equivalent to the previous "streaming mode". The
> community decided to get rid of this distinction, because it was
> confusing to users.
>
The new default is equivalent to the previous "streaming mode". The
community decided to get rid of this distinction, because it was
confusing to users.
The difference between "streaming mode" and "batch mode" was how
Flink's managed memory was allocated, either lazily when required
('streaming mo
Hi Fabian,
Previously when using flink 0.9-0.10 we start the cluster with streaming
mode or batch mode. I see that this one is gone on Flink 1.00 snapshot ? So
this one has already taken care of the flink and optimize by runtime >
On Mon, Feb 22, 2016 at 5:26 PM, Fabian Hueske wrote:
> Hi Welly
Hi Fabian,
Thanks a lot for your response.
- How many task managers do you start? I assume more than one TM per
machine given that you assign only 4GB of memory out of 128GB to each TM.
Currently what we have done is start a 1 TM per machine with number of task
slot 48.
- What is the maximum pa
Hi Welly,
I have to correct the formula I posted before:
taskmanager.network.numberOfBuffers: p ^ 2 * t * 4
p is NOT the parallelism of the job, BUT the number of slots of a task
manager.
So if you configure one TM for each machine with 48 slots, you get:
48^2 * 16 * 4 = 147.456 buffers, with 3
Hi Welly,
sorry for the late response.
The number of network buffers primarily depends on the maximum parallelism
of your job.
The given formula assumes a specific cluster configuration (1 task manager
per machine, one parallel task per CPU).
The formula can be translated to:
taskmanager.network
Hi All,
We are trying to running our job in cluster that has this information
1. # of machine: 16
2. memory : 128 gb
3. # of core : 48
However when we try to run we have an exception.
"insufficient number of network buffers. 48 required but only 10 available.
the total number of network buffers
10 matches
Mail list logo
