Re: Autoscaling vs backpressure

2020-05-07 Thread Arvid Heise
Hi Manish, while you could use backpressure and the resulting consumer lag to throttle the source and keep processing lag to a minimum, I'd personally see only very limited value. It assumes that you have an architecture where you can influence the input rate, which is probably only true if you ge

Re: Autoscaling vs backpressure

2020-05-06 Thread Aljoscha Krettek
I'd say the two can't be considered equivalent because the back pressure does not "reach" back into the source system. It only goes as far back as the Flink source. So if the outside system produces data to fast into the queue from which Flink is reading this input would keep piling up. Best,