Subject: Re: Regarding The Kafka Offset Management Issue In Direct Stream
Approach.
Questions about Spark-kafka integration are better directed to the Spark
user mailing list.
I'm not 100% sure what you're asking. The spark createDirectStream api
will not store any offsets internally,
Questions about Spark-kafka integration are better directed to the Spark
user mailing list.
I'm not 100% sure what you're asking. The spark createDirectStream api
will not store any offsets internally, unless you enable checkpointing.
On Sun, Nov 1, 2015 at 10:26 PM, Charan Ganga Phani Adabala
Hi All,
We are working in Apache spark with Kafka integration, in this use case we are
using DirectStream approach. we want to avoid the data loss in this approach
for actually we take offsets and saving that offset into MongoDB.
We want some clarification is Spark stores any offsets internally,
Questions about spark's kafka integration should probably be directed to
the spark user mailing list, not this one. I don't monitor kafka mailing
lists as closely, for instance.
For the direct stream, Spark doesn't keep any state regarding offsets,
unless you enable checkpointing. Have you read
Hi All,
We are working in Apache spark with Kafka integration, in this use case we are
using DirectStream approach. we want to avoid the data loss in this approach
for actually we take offsets and saving that offset into MongoDB.
We want some clarification is Spark stores any offsets internally,
