Re: How to Rate limit number of messages while writing Spark DataFrame to kafka topic

You cannot throttle via configs producer side, but maybe Kafka quotas can help? -Matthias On 11/7/24 2:53 AM, Divyanshu kumar wrote: 0 I have a spark dataframe which I am writing to Kafka topic using the below script. val ProducerConfig: Ma

How to Rate limit number of messages while writing Spark DataFrame to kafka topic

0 I have a spark dataframe which I am writing to Kafka topic using the below script. val ProducerConfig: Map[String, String] = Map( s"kafka.${ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG}" -> Integer.MAX_VALUE.toString, s"kafka.${ProducerCon