Re: Trying to understand KafkaConsumer_records_lag_max

2018-04-16 Thread Julio Biason
Hi Gordon (and list), Yes, that's probably what's going on. I got another message from 徐骁 which told me almost the same thing -- something I completely forgot (he also mentioned auto.offset.reset, which could be forcing Flink to keep reading from the top of Kafka instead of trying to go back and r

Re: Trying to understand KafkaConsumer_records_lag_max

2018-04-15 Thread Tzu-Li (Gordon) Tai
Hi Julio, I'm not really sure, but do you think it is possible that there could be some hard data retention setting for your Kafka topics in the staging environment? As in, at some point in time and maybe periodically, all data in the Kafka topics are dropped and therefore the consumers effectivel