Hi Xintong，
Thank you for your reply.
Do you mean you have 700 slots per TM or in total? How many TMs do you have?
And how many slots do you have per TM?
I have a Flink Cluster with 35 TMs,each TM has 16 slots.
cluster info: total TMs=35 ，total slots=560
Job info: request slot 400
It is after the job is fully initiated?
No,the job can't init.
Topology
op1-hash->op2-hash->op3-hash->op4
|
|-hash->op5
op1 parallelism is 200
op2 parallelism is 400
op3 parallelism is 400
op4 parallelism is 400
op5 parallelism is 400
Best Wishes
forideal
At 2020-03-20 15:20:07, "Xintong Song" <tonysong...@gmail.com> wrote:
Hi Forideal,
Do you mean you have 700 slots per TM or in total? How many TMs do you have?
And how many slots do you have per TM?
Also, when is the screenshot taken? It is after the job is fully initiated? It
seems you only need 1k+ network buffers.
Thank you~
Xintong Song
On Fri, Mar 20, 2020 at 12:07 PM forideal <fszw...@163.com> wrote:
Hi community
This parameter makes me confused.
| taskmanager.network.numberOfBuffers |
700000
|
In my job, i use 700 slots, but ,i have to set the this parameter to
700000.If not,i will get a exception.
java.io.IOException: Insufficient number of network buffers: required 700,
but only 1 available. The total number of network buffers is currently set to
80000 of 32768 bytes each. You can increase this number by setting the
configuration keys 'taskmanager.network.memory.fraction',
'taskmanager.network.memory.min', and 'taskmanager.network.memory.max'.
But actually this waste too many resource.
Memory Segments
| Type | Count |
| Available | 698,838 |
|
Total
| 700,000 |
| Direct | 700,103 | 21.4 GB | 21.4 GB |
| Mapped | 0 | 0 B | 0 B |
Best Wishes
forideal
