Further if there is are metrics that allows us to chart delays per pipe on n/w buffers, that would be immensely helpful.
On Fri, Jul 6, 2018 at 10:02 AM, Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Awesome, thank you for pointing that out. We have seen stability on pipes > where previously throttling the source ( rateLimiter ) was the only way out. > > https://github.com/apache/flink/blob/master/flink-core/ > src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L291 > > This though seems to be a cluster wide setting. Is it possible to do this > at an operator level ? Does this work with the pipe level configuration > per job ( or has that been deprecated ) > > On Thu, Jul 5, 2018 at 11:16 PM, Zhijiang(wangzhijiang999) < > wangzhijiang...@aliyun.com> wrote: > >> Hi Vishal, >> >> Before Flink-1.5.0, the sender tries best to send data on the network >> until the wire is filled with data. From Flink-1.5.0 >> the network flow control is improved by credit-based idea. That mea >> ns the sender transfers data based on how many buffers >> avaiable on receiver side, so there will be no data accumulated on the wire. >> From >> this point, the in-flighting data is less than before. >> >> Also you can further limit the in-flighting data by controling the number >> of credits on receiver side, and the related parameters are >> taskmanager.network.memory.buffers-per-channel and >> taskmanager.network.memory.floating-buffers-per-gate. >> >> If you have other questions about them, let me know then i can explain >> for you. >> >> Zhijiang >> >> ------------------------------------------------------------------ >> 发件人：Vishal Santoshi <vishal.santo...@gmail.com> >> 发送时间：2018年7月5日(星期四) 22:28 >> 收件人：user <user@flink.apache.org> >> 主 题：Limiting in flight data >> >> "Yes, Flink 1.5.0 will come with better tools to handle this problem. >> Namely you will be able to limit the “in flight” data, by controlling the >> number of assigned credits per channel/input gate. Even without any >> configuring Flink 1.5.0 will out of the box buffer less data, thus >> mitigating the problem." >> >> I read this in another email chain. The docs ( may be you can point me to >> them ) are not very clear on how to do the above. Any pointers will be >> appreciated. >> >> Thanks much. >> >> >> >
