Hi Team, I am using spark to read from S3 and write to Kafka.
Spark Version: 3.1.2 Scala Version: 2.12 Spark Kafka connector: spark-sql-kafka-0-10_2.12 I want to throttle kafka producer. I tried using *linger.ms <http://linger.ms>* and *batch.size* config but I can see in *ProducerConfig: ProducerConfig values* at runtime that they are not being set. Is there something I am missing? Is there any other way to throttle kafka writes? *dataset.write().format("kafka").options(options).save();* Regards, Abhishek Singla
