Sorry I missed this. Did you make sure that you add "kafka." as prefix on kafka side config when specifying Kafka source/sink option?
On Mon, Feb 24, 2025 at 10:31 PM Abhishek Singla < abhisheksingla...@gmail.com> wrote: > Hi Team, > > I am using spark to read from S3 and write to Kafka. > > Spark Version: 3.1.2 > Scala Version: 2.12 > Spark Kafka connector: spark-sql-kafka-0-10_2.12 > > I want to throttle kafka producer. I tried using *linger.ms > <http://linger.ms>* and *batch.size* config but I can see in *ProducerConfig: > ProducerConfig values* at runtime that they are not being set. Is there > something I am missing? Is there any other way to throttle kafka writes? > > *dataset.write().format("kafka").options(options).save();* > > Regards, > Abhishek Singla > > > > >
