implicit default minimum retention size per partition is 4GB.

Hi team, For default broker configuration, replica.lag.max.messages is 4000 and message.max.bytes is 1Mb. In the extreme case, the follower(s) could lag by 4000 messages. The leader must save at least 4000 messages to allow follower(s) to catch up. So the minimum retention size is 4000Mb=4Gb. I

RE: monitoring followers' lag

Thanks, Jun. I will try it out. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Tuesday, September 10, 2013 11:01 AM To: users@kafka.apache.org Subject: Re: monitoring followers' lag Have you looked at the updated docs in http://kafka.apache.org/document

Re: Mirror maker doesn't replicate new topics

Thanks Guozhang, 1, 2, 3 all are true. We are using default value 200 for batch.num.messages and 5000ms queue.buffering.max.ms. I believe it should batch either if batch.num.messages is reached or queue.buffering.max.ms is reached. I see log message "5000ms elapsed , Queue time reached. Sending.

Re: Failover for Zookeeper and Kafka

For asynchronous mirroring, you can take a look at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29 Thanks, Jun On Mon, Sep 9, 2013 at 11:30 PM, Xuyen On wrote: > Hi Joe, > > Yes I think we'd need to setup 3 zookeeper nodes in order to support a > single nod

Re: implicit default minimum retention size per partition is 4GB.

Monitoring the lag in bytes makes sense. The only difficulty is currently, the high watermark in the leader is represented in logical message offset, not the byte offset. For now, you will have to do the bytes to messages translation yourself. As for setting replica.lag.max.messages, you can obser

RE: implicit default minimum retention size per partition is 4GB.

In a stress test, 100K 1Mb messages (100Gb in size) are published (our bandwidth is limited). As our retention size is 3G which is smaller than the required default minimum retention size (4G), we noticed 20K messages were missing. After increasing "num.replica.fetchers" to 2, no more message

Re: implicit default minimum retention size per partition is 4GB.

You at least have to retain data long enough for the followers to be able to copy it from the leader. As Jun mentioned, you can observe the max lag in the follower and set replica.lag.max.messages to be a little higher than that. The high watermark is incremented to point to the logical offset of t

Re: Mirror maker doesn't replicate new topics

Oh got it. Did you set auto.offset.reset = smallest or largest? If it is largest it could be due to this bug: https://issues.apache.org/jira/browse/KAFKA-1006 Guozhang On Tue, Sep 10, 2013 at 10:09 AM, Rajasekar Elango wrote: > Hi Guozhang , > > 1) When I say "I send messages to new topic" ->

Re: Mirror maker doesn't replicate new topics

Hi Raja, When you say "I send messages to new topic" I guess you mean that you send messages to the source cluster right? It may be due to the fact that producers of mirror make have not catched up with the mirror maker consumer. When you say "I always see Handling 0 events" do you mean that you

RE: monitoring followers' lag

I have run JConsole. But I cannot find "kafka.server":name="([-.\w]+)-MaxLag", type="ReplicaFetcherManager" I am running 0.8 beta1. It seems ReplicaFetcherManager has not been added to this version. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Tuesd

Re: monitoring followers' lag

Have you looked at the updated docs in http://kafka.apache.org/documentation.html#monitoring ? Thanks, Jun On Tue, Sep 10, 2013 at 7:59 AM, Yu, Libo wrote: > Hi team, > > I wonder if anybody can give detailed instructions on how to monitor > the followers' lag by using JMX. Thanks. > > Regard

consumer partition rebalancing

I wasreviewingtheconsumer partition rebalancingalgorithm and had a fewrelatedquestions  * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:

monitoring followers' lag

Hi team, I wonder if anybody can give detailed instructions on how to monitor the followers' lag by using JMX. Thanks. Regards, Libo

Re: Mirror maker doesn't replicate new topics

Hi Guozhang , 1) When I say "I send messages to new topic" -> yes I am sending new messages to source cluster via console producer. 2) The log message "Handling 0 events" doesn't output topic name. But I would believe its for both old and new topics, because no other app is sending messages to sou

Re: consumer partition rebalancing

> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation > within a topic and 2) theconstraintthat a single consumer C(i) within a > consumer group C(g) must be the only reader of that partition: > * Are there ways to scale partition consumption if C(i) cannot keep > up?

Re: monitoring followers' lag

It should be "kafka.server":type="ReplicaFetcherManager",name="Replica-MaxLag" - can you confirm and mind updating the wiki if this is the case? Thanks, Joel On Tue, Sep 10, 2013 at 10:38 AM, Yu, Libo wrote: > I have run JConsole. But I cannot find > "kafka.server":name="([-.\w]+)-MaxLag", type