This should be handled by Flink. The system does flush records on
checkpoints and does not confirm a checkpoint before all flushes are acked
back.
Did you turn on checkpointing? Without that, Flink cannot give guarantees
for exactly the reason you outlined above.
On Wed, Mar 14, 2018 at 9:34 PM
Hi,
How are you checking that records are missing? Flink should flush to Kafka and
wait for all records to be flushed when performing a checkpoint.
Best,
Aljoscha
> On 13. Mar 2018, at 21:31, Chirag Dewan wrote:
>
> Hi,
>
> Still stuck around this.
>
> My understanding is, this is somethin
Hi,
Still stuck around this.
My understanding is, this is something Flink can't handle. If the batch-size of
Kafka Producer is non zero(which ideally should be), there will be in-memory
records and data loss(boundary cases). Only way I can handle this with Flink is
my checkpointing interval, w
Hi,
I am trying to use Kafka Sink 0.11 with ATLEAST_ONCE semantic and experiencing
some data loss on Task Manager failure.
Its a simple job with parallelism=1 and a single Task Manager. After a few
checkpoints(kafka flush's) i kill one of my Task Manager running as a container
on Docker Swarm.
