Yes, producing via kafka-clients using foreachPartition works as expected. Kafka Producer is initialised within call(Iterator<T> t) method.
The issue is with using kafka connector with Spark batch. The configs are not being honored even when they are being set in ProducerConfig. This means kafka records production rate cannot be controlled via kafka connector in Spark batch. This can lead to lag in in-sync replicas if they are not able to catch up and eventually kafka server failing writes it in-sync replicas count reduced the required in-sync replicas. Is there any way to solve this using kafka connector? Regards, Abhishek Singla
