Latency sounds high to me, maybe your JVMs are GC'ing a lot? Ryanne
On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinov...@gmail.com wrote: > Hi expert, > > > I would ask you some guidelines, web-pages or comments regarding my > use-case. > > > *Requirements*: > > - 2000+ producers > > - input rate 600k messages/s > > - consumers must write in 3 different databases (so i assume 3 consumer > groups) at 600k messages/s overall (200k messages/s/database) > > - latency < 500ms between producers and databases > > - good availability > > - Possibility to process messages before to send them to the databases > (Kafka stream? Of course in HA. Docker? Marathon?) > > - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly > required), latency has higher priority > > - record size: 100-1000 > > > *Resources*: > > brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s) > > producers -> brokers -> consumers ( Bandwidth: 1 Gbps ) > > > *My* *configuration*: > > 3 brokers > > 6 partition (without replication in order to minimize latency) > > ack = 0 (missing data is tolerate) > > batch.size = 1024 (with 8196 the throughput is max) > > producers -> compression.type=none > > > > I did test using kafka-producer-perf-test.sh and > kafka-consumer-perf-test.sh and i have a good throughput (500-600k > messages/s using 3 producers and 3 consumers) but i would improve > latency (0.3-2 sec) or features I'm not still considering. > > > I thank you in advance. > > Cheers, > > > Gioacchino > >
