Hi,
Generally, max production throughput will be calculated as follows:
- max_requests_per_sec = 1000 / request_latency_ms * 5
(max.in.flight.requests)
- max_throughput = max_requests_per_sec * messages-per-request
So I suggest identifying which is the bottleneck first, req-latency or
too-small-batch.
You can observe these from producer metrics.
2024年11月25日(月) 22:46 giri mungi <girimung...@gmail.com>:
> Hi Team,
>
> I am currently pushing 50,000 records to a Kafka producer, but the process
> is taking approximately 5 minutes to complete.
>
> Could you please assist me in optimizing the Kafka producer configurations
> to reduce the processing time? Any recommendations or best practices would
> be greatly appreciated.
> My configurations:
>
> props.put(ProducerConfig.ACKS_CONFIG, "all");
> props.put(ProducerConfig.BATCH_SIZE_CONFIG, 100000);
> props.put(ProducerConfig.LINGER_MS_CONFIG, 50);
> props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
> props.put("buffer.memory", 67108864);
> props.put("retries", 3);
> props.put("retry.backoff.ms", 5000);
> props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
> props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "10000");
>
> Producer<String, String> producer = new KafkaProducer<>(props);
>
>
> Thankyou,
> Giri
>
--
========================
Okada Haruki
ocadar...@gmail.com
========================
