Thanks Piotrek for the references. Cheers. Hemant
On Wed, Apr 14, 2021 at 7:18 PM Piotr Nowojski <pnowoj...@apache.org> wrote: > Hi, > > Depending how you configured your FlinkKafkaSource, but you can make the > source to commit consumed offsets back to Kafka. So one way to examine > them, would be to check those offsets in Kafka (I don't know how, but I'm > pretty sure there is a way to do it). > > Secondly, if you want to examine Flink's checkpoint state you can use > State Processor API to do that [1]. As far as I know you could hook up your > checkpointed data to Table API/SQL and use SQL to query/analyse the state. > > Best > Piotrek > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html > > śr., 14 kwi 2021 o 11:25 bat man <tintin0...@gmail.com> napisał(a): > >> Hi All, >> >> Is there any way I can inspect/query the checkpointed data. Scenario is >> like this - >> >> We have a high volume of data coming in the data stream pipeline for >> which kafka is source, in case if fails bcoz of bad data I want to >> analyse the data which caused the issue. It could be that some data source >> starts sending bad data so I want to go in kafka to that particular offset >> and do some analysis before I start the job with checkpointed data. >> >> Can anyone suggest how this can be achieved. >> >> Thanks, >> Hemant >> >> >>
