Re: Flink Kafka Consumer Throughput reduces over time

2020-04-01 Thread Piotr Nowojski
Hi, I haven’t heard about Flink specific problem like that. Have you checked that the records are not changing over time? That they are not for example twice as large or twice as heavy to process? Especially that you are using rate limiter with 12MB/s. If your records grew to 60KB in size, that

Flink Kafka Consumer Throughput reduces over time

2020-03-31 Thread Arpith techy
Currently I've Flink consumer with following properties, Flink consumes record at around 400 messages/sec at start of program but later on as numBuffersOut exceeds 100, data rate falls to 200messages/sec. I've set parallelism to only 1, it's Avro based consumer and checkpointing is disabled. D