Good to hear. Answers to your questions are inline. I think we should move the discussion to the dev@flink.a.o list though if there are more questions.
On Mon, Aug 10, 2015 at 9:29 AM, wangzhijiang999 <wangzhijiang...@aliyun.com > wrote: > Hi Ufuk, > > > Thank you for your detail and clear explaination! I reviewed the > code based on your info and got it clearly. > > > For local transfer: > > 1. > > When producer emits results to ResultPartition, it will request > Buffer from the pool. If there are no available Buffer, it will wait . > > Yes, the buffer request is blocking. > > 1. > > The ResultPartition of producer is the InputGate of Consumer, when > consumer read buffer from InputGate and deserializer the buffer, the Buffer > will be recycled ,so the producer can request Buffer again. > > The producer never requests a buffer from the input gate, but from its own pool. If the input gate is slow to recycle its buffers, the producer will become slow as well. But you are right that the producer will not be able to request a buffer if the input gate never recycles them. > > 1. > > So if the consumer slows, the Buffer in the ResultPartition of > producer can not be recycled quickly, resulting in producer has no > available Buffer to emit data. > 2. > > If the producer waits the available Buffer to emit, it can not process > elements and read next data from its InputGate, resulting in slowing the > producer. > > Yes. The task thread is both consuming and producing data. So if its blocked on the output side, the input is not consumed, which propagates back pressure further. > For remote transfer: > > The InputGate and ResultPartition for each task are separate > > 1. For producer the Buffer in ResultPartition will be recycled when > producer write them to the channel. > > 2. For consumer the Buffer will be recycled when consumer read it > from InputGate and deserializer it . The consumer need to request Buffer > when reading data from channel and put them to InputGate. > > 3. When consumer slows and there are no available Buffer , the > consumer will not read data from channel, so it will affect the producer > writing data to channel and the producer will have no available Buffer to > emit result at last resulting in slowing producer. > Yes but there is some logic involved to stop reading from the actual TCP channel to back pressure the remote producer. Essentially the selector read event is unsubscribed and on the output side there is Netty's watermark mechanism, which prevents further data from being written to the TCP channel.
