Re: metrics about how behind a replica is?

2014-12-31 Thread Jun Rao
In 0.8.2, we have a new java producer that allows to you specify a callback for each message to be sent. Thanks, Jun On Thu, Dec 18, 2014 at 12:07 PM, Xiaoyu Wang wrote: > @Jun, We can increase the number of resends, but the produce request may > still fail. > > For async producer, at the time

Re: the impact of partition number

2014-12-31 Thread Jun Rao
All writes to a single partition are synchronized on the broker. So, the more partitions the more parallelism (and better throughput) you have. Thanks, Jun On Mon, Dec 22, 2014 at 1:24 PM, Sa Li wrote: > Hi, All > > I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I >

Re: kafka 0.8.1.1 delay in replica data

2014-12-31 Thread Jun Rao
Is the system time in-sync btw the two hosts? Thanks, Jun On Tue, Dec 23, 2014 at 5:23 AM, chenlax wrote: > topic: > > Topic_test_data_1-10 > > leader broker data info: >-rw-rw-r-- 1 mqq mqq237728 Dec 23 19:51 000211076304.index > -rw-rw-r-- 1 mqq mqq 536875106 Dec 23 19:51 000

Re: Kafka 0.8.1.1 Leadership changes are happening very often

2014-12-31 Thread Jun Rao
A typical cause of frequent leadership changes is GC-induced soft failure. Do you see ZK session expiration on the broker? If so, you may want to enable GC log to see the GC time. Thanks, Jun On Tue, Dec 23, 2014 at 2:06 PM, Birla, Lokesh wrote: > > I was already using 4GB heap memory. I even

Re: Kafka JMX metrics is not a registered bean

2014-12-31 Thread Jun Rao
Not exactly sure, but you may have to escape the quotes. Thanks, Jun On Wed, Dec 24, 2014 at 3:29 AM, 马哲超 wrote: > Hello, > > I'm using cmdline-jmxclient to get Kafka JMX metrics values. When I run the > command below > > java -jar cmdline-jmxclient.jar - KAFKA_HOST:JMX_PORT > > it shows all r

Re: kafka consumer pull

2014-12-31 Thread Jun Rao
1. Kafka consumers support long poll. If there is no new message, a fetch request will block at the broker for up to max.wait.ms, 2. If you want the two consumers to each consume a full set of all messages, you should put them in different groups. On the other hand, if you want the two consumers to

Re: latency - how to reduce?

2014-12-31 Thread Jun Rao
What's your setting of log.flush.interval.messages on the broker? Thanks, Jun On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan wrote: > Hi, > I am using 0.8.1.1, and I have hundreds of msec latency at best and even > seconds at worst. > I have this latency both on production, (with peak load of 3