Re: Internal buffers

2016-06-02 Thread leon_mclare
Dear Ufuk, the wiki entry is exactly what i was looking for. I found it quite complicated to understand on a first attempt but i will dedicate some more time for it in the future. Thanks. Regards Leon 1. Jun 2016 13:06 by u...@apache.org: > There is this in the Wiki: > https://cwiki.apache.

Re: Internal buffers

2016-06-01 Thread Ufuk Celebi
There is this in the Wiki: https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks Buffers for data exchange come from the network buffer pool (by default 2048 * 32KB buffers). They are distributed to the running tasks and each logical channel between tasks needs at least one

Re: Internal buffers supervision and yarn vCPUs

2016-02-04 Thread Ufuk Celebi
> On 04 Feb 2016, at 12:02, Gwenhael Pasquiers > wrote: > > Ok thanks ! > > All that’s left is to wait then. If you have spare time and are working with the current snapshot version, it would be great to get some feedback on the pull request. :-) – Ufuk

RE: Internal buffers supervision and yarn vCPUs

2016-02-04 Thread Gwenhael Pasquiers
Ok thanks ! All that’s left is to wait then. B.R. From: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] On Behalf Of Stephan Ewen Sent: jeudi 4 février 2016 11:19 To: user@flink.apache.org Subject: Re: Internal buffers supervision and yarn vCPUs Concerning the first question: What you

Re: Internal buffers supervision and yarn vCPUs

2016-02-04 Thread Stephan Ewen
Concerning the first question: What you are looking for is backpressure monitoring. If a task cannot push its data to the next task, it is backpressured. This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578 We will try and get it merged soo

Re: Internal buffers supervision and yarn vCPUs

2016-02-04 Thread Robert Metzger
Hi Gwen, let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release. I think I removed back then because users were unable to define the number of vcores ind