Given your high throughput on the consumer side, you might consider adding more partitions than 6, so you can scale up beyond 6 consumers per group if need be.
Ryanne On Wed, Jan 9, 2019 at 10:37 AM Gioacchino Vino <gioacchinov...@gmail.com> wrote: > Hi Ryanne, > > > I just forgot to insert the "linger.ms=0" configuration. > > I got this result: > > > 50000000 records sent, 706793.701055 records/sec (67.41 MB/sec), 7.29 ms > avg latency, 1245.00 ms max latency, 0 ms 50th, 3 ms 95th, 197 ms 99th, > 913 ms 99.9th. > > > it's pretty good but I would like to improve it just a bit. > > Do you think using 6 partitions in a 3 broker cluster is a good choice? > > > Gioacchino > > > On 08/01/2019 18:52, Ryanne Dolan wrote: > > Latency sounds high to me, maybe your JVMs are GC'ing a lot? > > > > Ryanne > > > > On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinov...@gmail.com > > wrote: > > > >> Hi expert, > >> > >> > >> I would ask you some guidelines, web-pages or comments regarding my > >> use-case. > >> > >> > >> *Requirements*: > >> > >> - 2000+ producers > >> > >> - input rate 600k messages/s > >> > >> - consumers must write in 3 different databases (so i assume 3 consumer > >> groups) at 600k messages/s overall (200k messages/s/database) > >> > >> - latency < 500ms between producers and databases > >> > >> - good availability > >> > >> - Possibility to process messages before to send them to the databases > >> (Kafka stream? Of course in HA. Docker? Marathon?) > >> > >> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly > >> required), latency has higher priority > >> > >> - record size: 100-1000 > >> > >> > >> *Resources*: > >> > >> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s) > >> > >> producers -> brokers -> consumers ( Bandwidth: 1 Gbps ) > >> > >> > >> *My* *configuration*: > >> > >> 3 brokers > >> > >> 6 partition (without replication in order to minimize latency) > >> > >> ack = 0 (missing data is tolerate) > >> > >> batch.size = 1024 (with 8196 the throughput is max) > >> > >> producers -> compression.type=none > >> > >> > >> > >> I did test using kafka-producer-perf-test.sh and > >> kafka-consumer-perf-test.sh and i have a good throughput (500-600k > >> messages/s using 3 producers and 3 consumers) but i would improve > >> latency (0.3-2 sec) or features I'm not still considering. > >> > >> > >> I thank you in advance. > >> > >> Cheers, > >> > >> > >> Gioacchino > >> > >> >
