Hey Manu, I'm not aware of a benchmark on 10GbE. I'd love to see that though. Diving into the results may help us find bottlenecks hidden by the slower network.
Can you figure out where the bottleneck is in your test? I assume this is a single producer and consumer instance and you are using the new producer as in those benchmarks? This can be slightly tricky as it can be cpu or I/O on either the clients or the brokers. You basically have to look at top, iostat, and the jmx metrics for clues. The producer has good metrics that explain whether it is spending most of its time waiting or sending data. Not sure if there is a similar diagnostic for the consumer. -Jay On Tue, Nov 18, 2014 at 5:10 AM, Manu Zhang <owenzhang1...@gmail.com> wrote: > Hi all, > > I have been trying out kafka benchmarks described in Jay's > benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > < > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machine > >. > I'm able to get similar results on a 4-node GbE network whose in-bytes > could be saturated at 120MB/s. However, on a 4-node, 10GbE network, I can > not get in-bytes higher than 150MB/s. *Has anyone benchmarked kafka on a > 10GbE network ? Any rule of thumb on 10GbE network for configurations of > broker, producer and consumer ? * > > My kafka version is 0.8.1.1 and I've created a topic with 8 partitions with > 1 replica distributed evenly among the 4 nodes. Message size is 100 bytes. > I use all the default kafka settings. > My cluster has 4 nodes, where each node has 32 cores, 128MB RAM and 3 disks > for kafka. > > I've tried increasing message size to 1000 bytes which improved producer's > throughput but not consumer's. > > > Thanks, > Manu >
