Hi Eyal, Could you explain your job a bit more? Did you increase the parallelism of your job? What does it do? Does it perform any time based operations? How do you measure the processing rate?
Best, Dawid On 10/05/2020 21:18, Chen Qin wrote: > Hi Eyal, > > It’s unclear what warmup phase does in your use cases. Usually we see > Flink start consume at high rate and drop to a point downstream can > handle. > > Thanks > Chen > >> On May 10, 2020, at 12:25 AM, Eyal Pe'er <eyal.p...@startapp.com >> <mailto:eyal.p...@startapp.com>> wrote: >> >> Hi all, >> Lately I've added more resources to my Flink cluster which required a >> restart of all apps. >> From the cluster side, the only change I made, is to add more task slots. >> On the cluster I have a streaming app that consumes from Kafka and >> sinks to files. >> I noticed that since the restart, the applications "warmup" has >> impacted dramatically. >> Before, the change it took few minutes for the app to start and >> consume normally (from my point of view, normally is a stable rate) - >> from 0 to 16K events per second in 4 minutes. >> Now, after the change, it takes hours till it stabilizes on the >> normal processing rate- from 0 to 12K events per second in 3 hours. >> The data source behavior hasn’t changed (same incoming rate, >> partitions, servers etc.). >> I am aware to the backpressure mechanism in Flink, but it seems like >> it works too slow here. >> Is there a way to speed or control it? >> >> Thanks a lot >> Eyal Peer >
signature.asc
Description: OpenPGP digital signature
