For my use case, I need to figure out the lag within the Java consumer itself that is consuming some topic. Ideally, the consumer application would monitor the lag every minute or so and take some action on its own if consumer falls behind (i.e. spin up more threads to process records - my use case does not care about record order). For our purpose, it is OK if lag information is bit stale.
* AFAIK, Java KafkaConsumer APIs do not seem to expose any information directly based on which I can figure out the lag within my Java consumer application. * It seems that, alternatively, I can create separate new KafkaConsumer (or possibly use existing KafkaConsumer my consumer application is using), seek to end, and call "position()" API to figure out the end offsets for all partitions I am interested in. Since within my consumer application, I already know which offset consumer is at, I can figure out the lag. However, for this one basic information, this solution is bit more involved for our framework for various reasons. Also, it requires one to either use separate KafkaConsumer just to figure out lag or possibly re-use same KafkaConsumer our consumer application is using, but somehow implement the logic of forwarding to end to find out end offset and resetting back (not sure if it's feasible yet without issues). * If I am not mistaken, looking at KafkaConsumer code itself, "high watermark" information seems to be already there in FetchResponse (FetchResponse.PartitionData). It's just that it is not exposed. Is there any way to retrieve this information somehow in my consumer application? In general, wouldn't it be useful for consumer application to have lag information available with simple API call? Thanks, M
