This should be handled by Flink. The system does flush records on
checkpoints and does not confirm a checkpoint before all flushes are acked
back.
Did you turn on checkpointing? Without that, Flink cannot give guarantees
for exactly the reason you outlined above.
On Wed, Mar 14, 2018 at 9:34 PM
Hi,
How are you checking that records are missing? Flink should flush to Kafka and
wait for all records to be flushed when performing a checkpoint.
Best,
Aljoscha
> On 13. Mar 2018, at 21:31, Chirag Dewan wrote:
>
> Hi,
>
> Still stuck around this.
>
> My understanding is, this is somethin
Hi,
Still stuck around this.
My understanding is, this is something Flink can't handle. If the batch-size of
Kafka Producer is non zero(which ideally should be), there will be in-memory
records and data loss(boundary cases). Only way I can handle this with Flink is
my checkpointing interval, w
