Hi, Structured Streaming stores offsets only in HDFS compatible filesystems. Kafka and S3 are not such. Custom offset storage was only an option in DStreams.
G On Wed, Sep 30, 2020 at 9:45 AM Siva Samraj <samraj.mi...@gmail.com> wrote: > Hi all, > > I am using Spark Structured Streaming (Version 2.3.2). I need to read from > Kafka Cluster and write into Kerberized Kafka. > Here I want to use Kafka as offset checkpointing after the record is > written into Kerberized Kafka. > > Questions: > > 1. Can we use Kafka for checkpointing to manage offset or do we need to > use only HDFS/S3 only? > > Please help. > > Thanks > >
