Re: Get consumed Kafka offsets from Flink kafka source

2021-04-14 Thread bat man
Thanks Piotrek for the references. Cheers. Hemant On Wed, Apr 14, 2021 at 7:18 PM Piotr Nowojski wrote: > Hi, > > Depending how you configured your FlinkKafkaSource, but you can make the > source to commit consumed offsets back to Kafka. So one way to examine > them, would be to check those off

Re: Get consumed Kafka offsets from Flink kafka source

2021-04-14 Thread Piotr Nowojski
Hi, Depending how you configured your FlinkKafkaSource, but you can make the source to commit consumed offsets back to Kafka. So one way to examine them, would be to check those offsets in Kafka (I don't know how, but I'm pretty sure there is a way to do it). Secondly, if you want to examine Flin

Get consumed Kafka offsets from Flink kafka source

2021-04-14 Thread bat man
Hi All, Is there any way I can inspect/query the checkpointed data. Scenario is like this - We have a high volume of data coming in the data stream pipeline for which kafka is source, in case if fails bcoz of bad data I want to analyse the data which caused the issue. It could be that some data s