RE: Consumers re-reading data issue

2012-12-13 Thread Sybrandy, Casey
Jun, Funny you should ask...more data was loaded last night and no data was sent to the brokers since Dec. 11. The broker that I thought was the issue has not appeared since I removed the topic from it, so I'm guessing that's not the problem. Some other info that may be useful: We have over

Re: setting up 0.8

2012-12-13 Thread Jun Rao
John, Not yet. Could you try building it on a different machine? Thanks, Jun On Wed, Dec 12, 2012 at 10:59 PM, John Wang wrote: > Hi Jun: > > Is there a pre-built distribution that I can download? I am having > trouble with maven with my machine. > > -John > > On Thu, Dec 6, 2012 at 5:1

Re: Consumers re-reading data issue

2012-12-13 Thread Jun Rao
What's the ZK version? You should use 3.3.4 or above, which fixed some bugs that could cause rebalance to fail. Also, a typical reason for many rebalancing is the consumer side GC. If so, you will see logs saying sth like "expired session" for ZK. Thanks, Jun On Thu, Dec 13, 2012 at 5:54 AM, Sy

Re: How fair should I expect rebalancing to be?

2012-12-13 Thread Jun Rao
Do you see "fail to rebalance after 4 tries" in the worker not getting any data? If so, what's the ZK version? You should use 3.3.4 or above, which fixed some bugs that could cause rebalance to fail. Thanks, Jun On Wed, Dec 12, 2012 at 11:17 PM, David Ross wrote: > Hello, > > I am trying to di

RE: Consumers re-reading data issue

2012-12-13 Thread Sybrandy, Casey
The server is version 3.3.4. As for the client, I'm just using whatever Kakfa 0.7.0 came with. Not sure if that makes a different. If it does, I guess upgrading is the way to go. As for the GC part, I'll have to look into that some more. Thanks. -Original Message- From: Jun Rao [mai

RE: Design questions and your opinion and suggestions are appreciated

2012-12-13 Thread Jamie Wang
Thanks for the pointers. To clarify: my question 1) actually is referring to if I can dynamically create/add new partition to an existing topic on an existing broker? For pointer 2) you seem to suggest a newer version of the broker will have a different behavior. what would this new behavior l

Re: How fair should I expect rebalancing to be?

2012-12-13 Thread David Ross
Looking into the "fail to rebalance" messages. We do have zk 3.3.4. Could a higher number of partitions be the cause? On Thu, Dec 13, 2012 at 8:03 AM, Jun Rao wrote: > Do you see "fail to rebalance after 4 tries" in the worker not getting any > data? If so, what's the ZK version? You should use

Broker Interface Binding

2012-12-13 Thread Matan Amir
Hi All, I've scanned the code a little (0.7.2) but couldn't find a parameter to force the broker to bind to a specific network interface rather than listen on all. Was this added to 0.8? I was hoping for something like ZK's clientPortAddress configuration parameter. Cheers, Matan

Subscription Issue.

2012-12-13 Thread Jason Huang
Hello, I am interested in learning more about Kafka. I just sent a subscription request to kafka-users-subscr...@incubator.apache.org (as indicated at http://incubator.apache.org/kafka/contact.html) but this email address seems to be incorrect. Could someone help to add me to the mail list? tha

RE: Subscription Issue.

2012-12-13 Thread Jamie Wang
The email below from Jun will should help you on the subscription. On Wed, Dec 5, 2012 at 1:18 PM, Jun Rao wrote: > We recently graduated to a top-level Apache project and have new > mailing lists. You need to subscribe to > dev-subscr...@kafka.apache.org > users-subscr...@kafka.apache.org

Consumer does not use broker.list property?

2012-12-13 Thread Subhash Agrawal
Hi All, I just noticed that consumer need to have zookeeper URL. It does not use broker.list property. Can somebody please confirm if this is the case. Thanks Subhash A.

Re: Consumers re-reading data issue

2012-12-13 Thread Jun Rao
Try the 0.7.1 Kafka jar, which fixed some consume side bugs. Thanks, Jun On Thu, Dec 13, 2012 at 8:14 AM, Sybrandy, Casey < casey.sybra...@six3systems.com> wrote: > The server is version 3.3.4. As for the client, I'm just using whatever > Kakfa 0.7.0 came with. Not sure if that makes a differ

Re: Design questions and your opinion and suggestions are appreciated

2012-12-13 Thread Jun Rao
For 2), the next Kafka release will support replication. So a partition is still available when a single broker is down. For 1), in 0.7, partitions are automatically added to new brokers. In 0.8, you will need to run a command to change # partitions. Thanks, Jun On Thu, Dec 13, 2012 at 10:39 AM

Re: How fair should I expect rebalancing to be?

2012-12-13 Thread Jun Rao
Shouldn't be. Jun On Thu, Dec 13, 2012 at 11:36 AM, David Ross wrote: > Looking into the "fail to rebalance" messages. We do have zk 3.3.4. Could a > higher number of partitions be the cause? > > On Thu, Dec 13, 2012 at 8:03 AM, Jun Rao wrote: > > > Do you see "fail to rebalance after 4 tries"

Re: Broker Interface Binding

2012-12-13 Thread Jun Rao
In 0.7, you can set hostname in the broker config. Thanks, Jun On Thu, Dec 13, 2012 at 11:50 AM, Matan Amir wrote: > Hi All, > > I've scanned the code a little (0.7.2) but couldn't find a parameter to > force the broker to bind to a specific network interface rather than listen > on all. Was

Re: Consumer does not use broker.list property?

2012-12-13 Thread Jun Rao
Yes. Jun On Thu, Dec 13, 2012 at 3:43 PM, Subhash Agrawal wrote: > Hi All, > > I just noticed that consumer need to have zookeeper URL. It does not use > broker.list property. > > Can somebody please confirm if this is the case. > > Thanks > Subhash A. >