Re: Spark Streaming Checkpoint and Exactly Once Guarantee on Kafka Direct Stream

In either case, end to end exactly once guarantee can only be ensured only if the output sink is updated transactionally. The engine has to re execute data on failure. Exactly once guarantee means that the external storage is updated as if each data record was computed exactly once. That's why you

Re: Spark Streaming Checkpoint and Exactly Once Guarantee on Kafka Direct Stream

Thanks TD. In pre-structured streaming, exactly once guarantee on input is not guaranteed. is it? On Tue, Jun 6, 2017 at 4:30 AM, Tathagata Das wrote: > This is the expected behavior. There are some confusing corner cases. > If you are starting to play with Spark Streaming, i highly recommend >

Re: Spark Streaming Checkpoint and Exactly Once Guarantee on Kafka Direct Stream

This is the expected behavior. There are some confusing corner cases. If you are starting to play with Spark Streaming, i highly recommend learning Structured Streaming instead. On Mon, Jun 5, 2017 at 11:16 AM, anbuc