Re: High-level consumer load-balancing problem

The consumer load balancing logic today is pretty simple. It just tries to divide the partitions evenly among the consumers. It doesn't try to balance by load. Thanks, Jun On Thu, Nov 14, 2013 at 3:34 PM, hsy...@gmail.com wrote: > Hi, > > I have questions about the load balancing of kafka hig

Re: broker exception

0.8-beta1 should come with ZK 3.3.4. Thanks, Jun On Thu, Nov 14, 2013 at 11:09 AM, Yu, Libo wrote: > We are using the default one in 0.8-beta1. It should be 3.3.3. > According to the comment in the Jira, it is more like a zookeeper > configuration issue. I will make change to our configuratio

Multiple consumer connectors in process - lag metrics?

Hi, We are experimenting with having multiple consumer connectors running in the same process, under the same groupId (but with different topic filters). I'm wondering what the expected effect of this is with metrics, like ConsumerFetcherManager.-MaxLag It looks like in AbstractFetcherManager, t

High-level consumer load-balancing problem

Hi, I have questions about the load balancing of kafka high-level consumer Suppose I have 4 partition And the producer throughput to these 4 partitions are like this 01 23 10MB/s 10MB/s 1MB/s1MB/s 1kMsg/s,10kMs

Re: Leader not local

I think it should be some problems on the producer side, but without any logs it will be very hard to trouble shoot. Guozhang On Thu, Nov 14, 2013 at 11:02 AM, Yu, Libo wrote: > Unfortunately, we don't have access to producer log. But it seems very > obvious to me that producer > failed to fin

RE: broker exception

We are using the default one in 0.8-beta1. It should be 3.3.3. According to the comment in the Jira, it is more like a zookeeper configuration issue. I will make change to our configuration and keep monitoring it. Thanks. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gm

RE: Leader not local

Unfortunately, we don't have access to producer log. But it seems very obvious to me that producer failed to find out the correct leader as the warning messages occurred repetitively for a pretty long time span (more than 10 minutes). Regards, Libo -Original Message- From: Guozhang W

Re: broker exception

Are you using ZK 3.3.4? This seems to be caused by a bug in 3.3.3 and 3.3.0. https://issues.apache.org/jira/browse/ZOOKEEPER-1115 Thanks, Jun On Thu, Nov 14, 2013 at 5:25 AM, Yu, Libo wrote: > Hi team, > > This exception occurs regularly on our brokers. When it occurs, a broker > will lose i

Re: will this cause message loss?

Also if you use HEAD, you can create more partitions at runtime, you just need dynamic partitioner class I think On Thu, Nov 14, 2013 at 7:23 AM, Neha Narkhede wrote: > There is no way to delete topics in Kafka yet. You can add partitions to > existing topics, but you may have to use 0.8 HEAD si

Re: Leader not local

Hi Libo, Yes the producer should be start to find the correct leader upon just one send failures. Could you check if there are other exceptions on the producer log? Guozhang On Thu, Nov 14, 2013 at 8:23 AM, Yu, Libo wrote: > Hi team, > > I saw this line within a long time span in our logs for

Leader not local

Hi team, I saw this line within a long time span in our logs for the same topic and partition. [2013-11-14 11:13:41,647] WARN [KafkaApi-1] Produce request with correlation id 529240 from client on partition [mytopic,2] failed due to Leader not local for partition [mytopic,2] on broker 1 (kafka

Re: will this cause message loss?

There is no way to delete topics in Kafka yet. You can add partitions to existing topics, but you may have to use 0.8 HEAD since we have fixed a few bugs on the consumer. You can read about add partitions here https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.A

Re: Kafka cluster with lots of topics

Thanks for the replies. I don't think Kafka quite fits our use case, unfortunately. To abstractly answer Edward's question: in a system with lots of users, we were considering having a topic per user (such that an individual user can connect from a number of endpoints and receive events, including

will this cause message loss?

Hi team, We are using beta1. I am going to delete all topics and create them with more partitions. But I don't want to lose any messages. Assume the consumers are online all the time for the following steps. The consumer's auto.offset.reset is set to largest. 1 stop publishing to the brokers.

broker exception

Hi team, This exception occurs regularly on our brokers. When it occurs, a broker will lose its leader role but still in ISR. And running preferred-leader-election script may rebalance the leadership but in some cases it does not help. [2013-11-14 08:04:40,001] INFO Processed session terminatio