Re: Reserving Kafka offset in Flink after modifying app

2019-04-03 Thread shengjk1
Mapbe this page can help you https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/savepoints.html Best, Shengjk1 On 03/26/2019 09:51,Son Mai wrote: Hi Konstantin, Thanks for the response. What still concerned me is: Am I able to recover from checkpoints even if I change m

Re: Reserving Kafka offset in Flink after modifying app

2019-03-25 Thread Yun Tang
: Konstantin Knauf Cc: user Subject: Re: Reserving Kafka offset in Flink after modifying app Hi Konstantin, Thanks for the response. What still concerned me is: 1. Am I able to recover from checkpoints even if I change my program (for example, changing Filter and Map functions, data objects

Re: Reserving Kafka offset in Flink after modifying app

2019-03-25 Thread Son Mai
Hi Konstantin, Thanks for the response. What still concerned me is: 1. Am I able to recover from checkpoints even if I change my program (for example, changing Filter and Map functions, data objects, ..) ? I was not able to recover from savepoints when I changed my program. On Mon, Ma

Re: Reserving Kafka offset in Flink after modifying app

2019-03-25 Thread Konstantin Knauf
Hi Son, yes, this is possible, but your sink needs to play its part in Flink's checkpointing mechanism. Depending on the implementation of the sink you should either: * implemented *CheckpointedFunction *and flush all records to BigQuery in *snapshotState*. This way in case of a failure/restart o

Reserving Kafka offset in Flink after modifying app

2019-03-24 Thread Son Mai
Hello, I have a topic in Kafka that Flink reads from. I parse and write messages in this topic to BigQuery using streaming insert in batch of 500 messages using in CountWindow in Flink. *Problem*: I want to commit manually only when a batch was written successfully to Bigquery. *Reason:* I saw t