This is a good point, even though you mentioned that you also have latency issues locally. I just migrated a 3-node test cluster from m3.large instances to c4.xlarge instances (3-node ZK migrated from m3.medium to c4.large) in an EC2 placement group (better network IO and more consistent latencies), and both throughput and latencies improved dramatically. We're deliberately using ack all with replication factor of 2 and only 1 partition, and even with this we comfortably hit 10K messages/sec with sub ms latency, and 100K messages per sec with latencies in the 4-5ms range. On the m3.large cluster we maxed out at a couple of thousand messages/sec. And we haven't tuned any settings for performance (quite the opposite!), other than to write all logs to dedicated EBS volumes.
What instance types are you using, both for ZK and for Kafka? The lower end, general purpose instance types really aren't that spectacular, especially with regard to networking. And look into AWS placement groups. Cheers, Shannon On 19 Jan 2015 16:52, "Andrew Ehrlich" <and...@aehrlich.com> wrote: > What instance types are you using in EC2? Are the drives EBS? > > On 1/5/15 11:06 PM, Shlomi Hazan wrote: > >> Will do. What did you have in mind? just write a big file to disk and >> measure the time it took to write? maybe also read back? using specific >> API's? >> Apart from the local Win machine case, are you aware of any issues with >> Amazon EC2 instances that may be causing that same latency in production? >> Thanks, >> Shlomi >> >> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <j...@confluent.io> wrote: >> >> Not setting "log.flush.interval.messages" is good since the default gives >>> the best latency. Could you do some basic I/O testing on the local FS in >>> your windows machine to make sure the I/O latency is ok? >>> >>> Thanks, >>> >>> Jun >>> >>> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shl...@viber.com> wrote: >>> >>> Happy new year! >>>> I did not set "log.flush.interval.messages". >>>> I also could not find a default value in the docs. >>>> Could you explain about that? >>>> Thanks, >>>> Shlomi >>>> >>>> On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <j...@confluent.io> wrote: >>>> >>>> What's your setting of log.flush.interval.messages on the broker? >>>>> >>>>> Thanks, >>>>> >>>>> Jun >>>>> >>>>> On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shl...@viber.com> >>>>> >>>> wrote: >>> >>>> Hi, >>>>>> I am using 0.8.1.1, and I have hundreds of msec latency at best and >>>>>> >>>>> even >>>> >>>>> seconds at worst. >>>>>> I have this latency both on production, (with peak load of 30K >>>>>> >>>>> msg/sec, >>> >>>> replication = 2 across 5 brokers, acks = 1), >>>>>> and on the local windows machine using just one process for each of >>>>>> producer, zookeeper, kafka, consumer. >>>>>> Also tried batch.num.messages=1 and producer.type=sync on the local >>>>>> >>>>> machine >>>>> >>>>>> but saw no improvement. >>>>>> How can I push latency down to several millis, at least when running >>>>>> >>>>> local? >>>>> >>>>>> Thanks, >>>>>> Shlomi >>>>>> >>>>>> > >
