Hi Team,
I am currently pushing 50,000 records to a Kafka producer, but the process
is taking approximately 5 minutes to complete.
Could you please assist me in optimizing the Kafka producer configurations
to reduce the processing time? Any recommendations or best practices would
be greatly appreciated.
My configurations:
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 100000);
props.put(ProducerConfig.LINGER_MS_CONFIG, 50);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
props.put("buffer.memory", 67108864);
props.put("retries", 3);
props.put("retry.backoff.ms", 5000);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "10000");
Producer<String, String> producer = new KafkaProducer<>(props);
Thankyou,
Giri
