Hi Vishal, let me already point you towards the JIRA issue for the credit-based flow control: https://issues.apache.org/jira/browse/FLINK-7282
I'll have a look at the rest of this email thread tomorrow... Regards, Nico On 02/01/18 17:52, Vishal Santoshi wrote: > Could you please point me to any documentation on the "credit-based > flow control" approach.... > > On Tue, Jan 2, 2018 at 10:35 AM, Timo Walther <twal...@apache.org > <mailto:twal...@apache.org>> wrote: > > Hi Vishal, > > your assumptions sound reasonable to me. The community is currently > working on a more fine-grained back pressuring with credit-based > flow control. It is on the roamap for 1.5 [1]/[2]. I will loop in > Nico that might tell you more about the details. Until then I guess > you have to implement a custom source/adapt an existing source to > let the data flow in more realistic. > > Regards, > Timo > > [1] > http://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html > > <http://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html> > [2] https://www.youtube.com/watch?v=scStdhz9FHc > <https://www.youtube.com/watch?v=scStdhz9FHc> > > > Am 1/2/18 um 4:02 PM schrieb Vishal Santoshi: > > I did a simulation on session windows ( in 2 modes ) and let it > rip for about 12 hours > > 1. Replay where a kafka topic with retention of 7 days was the > source ( earliest ) > 2. Start the pipe with kafka source ( latest ) > > I saw results that differed dramatically. > > On replay the pipeline stalled after good ramp up while in the > second case the pipeline hummed on without issues. For the same > time period the data consumed is significantly more in the > second case with the WM progression stalled in the first case > with no hint of resolution ( the incoming data on source topic > far outstrips the WM progression ) I think I know the reasons > and this is my hypothesis. > > In replay mode the number of windows open do not have an upper > bound. While buffer exhaustion ( and data in flight with > watermark ) is the reason for throttle, it does not really > limit the open windows and in fact creates windows that reflect > futuristic data ( future is relative to the current WM ) . So if > partition x has data for watermark time t(x) and partition y for > watermark time t(y) and t(x) << t(y) where the overall watermark > is t(x) nothing significantly throttles consumption from the y > partition ( in fact for x too ) , the bounded buffer based > approach does not give minute control AFAIK as one would hope > and that implies there are far more open windows than the system > can handle and that leads to the pathological case where the > buffers fill up ( I believe that happens way late ) and > throttling occurs but the WM does not proceed and windows that > could ease the glut the throttling cannot proceed..... In the > replay mode the amount of data implies that the Fetchers keep > pulling data at the maximum consumption allowed by the open > ended buffer approach. > > My question thus is, is there any way to have a finer control of > back pressure, where in the consumption from a source is > throttled preemptively ( by for example decreasing the buffers > associated for a pipe or the size allocated ) or sleeps in the > Fetcher code that can help aligning the performance to have real > time consumption characteristics > > Regards, > > Vishal. > > > > > >
signature.asc
Description: OpenPGP digital signature
