Thanks Jay for the quick response. Yes, it's a single producer and consumer both configured with multiple threads but I'm not using the new producer. CPU is typically 50% utilized on client and merely used on broker. Disks aren't busy either as a lot of data are cached in memory. Would you please give a link for the producer metrics you are referring to ?
Thanks, Manu On Wed, Nov 19, 2014 at 2:39 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey Manu, > > I'm not aware of a benchmark on 10GbE. I'd love to see that though. Diving > into the results may help us find bottlenecks hidden by the slower network. > > Can you figure out where the bottleneck is in your test? I assume this is a > single producer and consumer instance and you are using the new producer as > in those benchmarks? > > This can be slightly tricky as it can be cpu or I/O on either the clients > or the brokers. You basically have to look at top, iostat, and the jmx > metrics for clues. The producer has good metrics that explain whether it is > spending most of its time waiting or sending data. Not sure if there is a > similar diagnostic for the consumer. > > -Jay > > On Tue, Nov 18, 2014 at 5:10 AM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > > Hi all, > > > > I have been trying out kafka benchmarks described in Jay's > > benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > < > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machine > > >. > > I'm able to get similar results on a 4-node GbE network whose in-bytes > > could be saturated at 120MB/s. However, on a 4-node, 10GbE network, I can > > not get in-bytes higher than 150MB/s. *Has anyone benchmarked kafka on a > > 10GbE network ? Any rule of thumb on 10GbE network for configurations of > > broker, producer and consumer ? * > > > > My kafka version is 0.8.1.1 and I've created a topic with 8 partitions > with > > 1 replica distributed evenly among the 4 nodes. Message size is 100 > bytes. > > I use all the default kafka settings. > > My cluster has 4 nodes, where each node has 32 cores, 128MB RAM and 3 > disks > > for kafka. > > > > I've tried increasing message size to 1000 bytes which improved > producer's > > throughput but not consumer's. > > > > > > Thanks, > > Manu > > >
