Hi, Julio: 1. Flink doesn't use subscribe because it needs to control partition assignment itself, which is important for implementing exactly once. 2. Can you share the versions you are using, including kafka, kafka client, flink? We are also use flink kafka consumer and we can monitor it correctly.
On Tue, Sep 4, 2018 at 3:09 AM Julio Biason <julio.bia...@azion.com> wrote: > Hey guys, > > We are trying to add external monitoring to our system, but we can only > get the lag in kafka topics while the Flink job is running -- if, for some > reason, the Flink job fails, we get no visibility on how big the lag is. > > (Besides that, the way Flink reports is not accurate and produces a lot of > -Inf, which I already discussed before.) > > While looking at the problem, we noticed that the FlinkKafkaConsumer never > uses `subscribe` to subscribe to the topics and that's why the values are > never stored back into Kafka, even when the driver itself does > `commitAsync`. > > Is there any reason for not subscribing to topics that I may have missed? > > -- > *Julio Biason*, Sofware Engineer > *AZION* | Deliver. Accelerate. Protect. > Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 > <callto:+5551996209291>*99907 0554* > -- Liu, Renjie Software Engineer, MVAD
