Hi,
Structured Streaming stores offsets only in HDFS compatible filesystems.
Kafka and S3 are not such.
Custom offset storage was only an option in DStreams.
G
On Wed, Sep 30, 2020 at 9:45 AM Siva Samraj wrote:
> Hi all,
>
> I am using Spark Structured Streaming (Version 2.3.2). I need to rea
Hi all,
I am using Spark Structured Streaming (Version 2.3.2). I need to read from
Kafka Cluster and write into Kerberized Kafka.
Here I want to use Kafka as offset checkpointing after the record is
written into Kerberized Kafka.
Questions:
1. Can we use Kafka for checkpointing to manage offset
