Hello, I have a topic in Kafka that Flink reads from. I parse and write messages in this topic to BigQuery using streaming insert in batch of 500 messages using in CountWindow in Flink.
*Problem*: I want to commit manually only when a batch was written successfully to Bigquery. *Reason:* I saw that Flink KafkaConsumer does not use offset committing to Kafka but uses its own checkpointing. I don't know how Flink checkpointing works and I'm worried that Flink's checkpointing does not solve my following situation: - let's say I have a Flink job running and processing a batch of 500 messages of Kafka offset 1000-1500. - I stopped this job before it saves to BigQuery and makes some modifications to the program. Savepoints did not work when I tried because it required the operators code does not change. What I want is when I start the modified app, it would start every time from offset 1000-1500 in Kafka because these messages have not been written to BigQuery. Is there any way to achieve this in Flink? Thanks, SM
