Hi Lu, I would say that if your application is stable and checkpoints do not timeout there is no immediate necessity to do anything. The fact that the consumer lag stays low means that you are able to keep up with the incoming data. That said, the fact that you observe "constant backpressure" with no consumer lag increase sounds a bit strange. Backpressure propagates upstream in Flink, meaning that a backpressured operator will, by design, eventually throttle data consumption if backpressure downstream is constant. Keep in mind that accidental backpressure is fine - its purpose is to prevent data loss if you have accidental input data bursts. You might find this video helpful if you want to investigate the source of backpressure and what to do with it: https://youtu.be/bhcFfS1-eDY?t=408
P.S: such questions are better kept to the user mailing list. Dev mailing list is solely for the purpose of discussing topics around Flink development. Best, Alex On Fri, 23 Jun 2023 at 06:55, Lu Niu <qqib...@gmail.com> wrote: > > For example, if a flink job reads from kafka do something and writes to > kafka. Do we need to take any actions when the job kafka consumer lag is > low or 0 but some tasks have constant backpressure? Do we need to increase > the parallelism or do some network tuning so that backpressure is constant > 0? If so, would that lead to resource overprovision? > > Or is it that only when kafka lag keeps increasing while backpressure is > happening at the same time, we need to take action? > > > Best > > Lu
