Re: kafka consumer not consuming messages

2014-02-12 Thread Arjun
I am sorry but could not locate the offset in the request log. I have turned on the debug for the logs but couldnt . Do you know any pattern with which i can look in. Thanks Arjun Narasimha Kota On Wednesday 12 February 2014 09:26 PM, Jun Rao wrote: Interesting. So you have 4 messages in the

Re: Possibly leaking socket on ReplicaFetcherThread

2014-02-12 Thread Jun Rao
This is fixed in https://issues.apache.org/jira/browse/KAFKA-1228 and will be included in 0.8.1 release. Thanks, Jun On Wed, Feb 12, 2014 at 6:28 PM, Priya Matpadi wrote: > Hello, > Is there any progress on this issue? We also experience socket leak in case > of network outage. > Thanks, > Pri

Re: Error while fetching metadata for partition [topic-pixel,1] (kafka.server.KafkaApis)

2014-02-12 Thread Jun Rao
Are those errors transient or persistent? Transient errors when a topic was first created are normal. Thanks, Jun On Wed, Feb 12, 2014 at 4:05 PM, David Montgomery wrote: > I just built a new kafka server 0.8.0. How I am getting the below in the > logs. What does it mean? > > > [2014-02-13

Re: Possibly leaking socket on ReplicaFetcherThread

2014-02-12 Thread Priya Matpadi
Further more, the problem is not just restricted to ReplicaFetcherThread. Kafka consumer server also leaks sockets due to SendThread using same code . See below stack trace: 2014-01-23 06:48:09,699 INFO [org.apache.zookeeper.ClientCnxn] (OurKafkaMessageFetcher-blah1-SendThread(pkafka3.our.com:218

Re: Possibly leaking socket on ReplicaFetcherThread

2014-02-12 Thread Priya Matpadi
Hello, Is there any progress on this issue? We also experience socket leak in case of network outage. Thanks, Priya On Fri, Jan 24, 2014 at 7:30 AM, Jun Rao wrote: > Thanks for find this out. We probably should disconnect on any exception. > Could you file a jira and perhaps attach a patch? > >

Re: Error while fetching metadata for partition [topic-pixel,1] (kafka.server.KafkaApis)

2014-02-12 Thread Guozhang Wang
I was asking if you could first verify if the topic named "topic-pixel" exists in Kafka using the kafka-topics tool. Guozhang On Wed, Feb 12, 2014 at 4:27 PM, David Montgomery wrote: > What does that mean? I have one zk and one kafka and one cient trying to > pull from kafka. > > Thanks > > >

Re: Error while fetching metadata for partition [topic-pixel,1] (kafka.server.KafkaApis)

2014-02-12 Thread David Montgomery
What does that mean? I have one zk and one kafka and one cient trying to pull from kafka. Thanks On Thu, Feb 13, 2014 at 8:16 AM, Guozhang Wang wrote: > It means the leader information has not propagated to the brokers, hence > when clients asked for the leaders of the interested partitions t

Re: Error while fetching metadata for partition [topic-pixel,1] (kafka.server.KafkaApis)

2014-02-12 Thread Guozhang Wang
It means the leader information has not propagated to the brokers, hence when clients asked for the leaders of the interested partitions this error is thrown. Have you created the topic manually before you try to produce/consume? Guozhang On Wed, Feb 12, 2014 at 4:05 PM, David Montgomery wrote

Re: Sync producers stuck waiting for 2 acks

2014-02-12 Thread Joel Koshy
> I checked the information in Zookeeper and found out that 2 of the brokers > are missing. The VMs with these brokers are not quite ... healthy (I cannot > find another definition for this situation). I checked the information about > replicas distribution and there are 3 replicas for each part

RE: Sync producers stuck waiting for 2 acks

2014-02-12 Thread Michael Popov
Thanks Joel! I found this configuration setting in "Producer Configs". I guess it means each producer sets this parameter as part of connection settings, like a number of acks. I checked the information in Zookeeper and found out that 2 of the brokers are missing. The VMs with these brokers are

Error while fetching metadata for partition [topic-pixel,1] (kafka.server.KafkaApis)

2014-02-12 Thread David Montgomery
I just built a new kafka server 0.8.0. How I am getting the below in the logs. What does it mean? [2014-02-13 00:01:43,034] ERROR [KafkaApi-1392248055] Error while fetching metadata for partition [topic-pixel,0] (kafka.server.KafkaApis) kafka.common.LeaderNotAvailableException: Leader not avail

Re: New Consumer API discussion

2014-02-12 Thread Jay Kreps
Ah, gotcha. -Jay On Wed, Feb 12, 2014 at 8:40 AM, Neha Narkhede wrote: > Jay > > Well none kind of address the common case which is to commit all > partitions. For these I was thinking just >commit(); > The advantage of this simpler method is that you don't need to bother about > partitions

Re: Sync producers stuck waiting for 2 acks

2014-02-12 Thread Joel Koshy
The request time out config is request.timeout.ms - defaults to 10 seconds - so the request should expire by then and return a response. Can you run the list-topics command on the topics you are sending to and make sure there are at least two replicas in ISR while you are running your producer test

Sync producers stuck waiting for 2 acks

2014-02-12 Thread Michael Popov
I am running a test deployment of Kafka 0.8. When I configure sync producers to expect 2 acks for each "write" request, some of the producers get stuck. It looks like broker's response is not delivered back. This happened with original Kafka performance tools and with a test tool built using a c

Re: where is my kafka ip address in ZK. I get nothing in a broker ID

2014-02-12 Thread David DeMaagd
That information is in that node, not under it (you want a get() instead of a get_children())... -- Dave DeMaagd | S'aite Reliability Engineering, Y'all ddema...@linkedin.com | 818 262 7958 (davidmontgom...@gmail.com - Thu, Feb 13, 2014 at 06:09:07AM +0800) > Hi, > > I am using kafka 8.0. > >

where is my kafka ip address in ZK. I get nothing in a broker ID

2014-02-12 Thread David Montgomery
Hi, I am using kafka 8.0. when I set up kafkaand I look at the broker id in ZK...this is what I get: #my python script print client.get_children('/brokers/ids/1391209402') [ ] should I not see the ip address? Thanks

RE: some brokers cannot register themselves with zookeeper

2014-02-12 Thread Yu, Libo
Thanks, Guozhang. We are using 0.8-beta1. Regards, Libo -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Tuesday, February 11, 2014 6:40 PM To: users@kafka.apache.org Subject: Re: some brokers cannot register themselves with zookeeper Hello Libo, Which Kafka v

Re: kafka consumer not consuming messages

2014-02-12 Thread Arjun Kota
Hi, No i havent changed the auto commit enable. That one message is the one which got earlier long time back(2 weeks back). After that i started working recently and things started behaving werid. I dont have the request log now, will check and let u know. Thanks Arjun narasimha k On Feb 12, 20

Re: New Consumer API discussion

2014-02-12 Thread Neha Narkhede
Jay Well none kind of address the common case which is to commit all partitions. For these I was thinking just commit(); The advantage of this simpler method is that you don't need to bother about partitions you just consume the messages given to you and then commit them This is already what t

Re: New Consumer API discussion

2014-02-12 Thread Neha Narkhede
Imran, Sorry I am probably missing something basic, but I'm not sure how a multi-threaded consumer would work. I can imagine its either: a) I just have one thread poll kafka. If I want to process msgs in multiple threads, than I deal w/ that after polling, eg. stick them into a blocking queue o

Re: kafka consumer not consuming messages

2014-02-12 Thread Jun Rao
Interesting. So you have 4 messages in the broker. The checkpointed offset for the consumer is at the 3rd message. Did you change the default setting of auto.commit.enable? Also, if you look at the request log, what's the offset in the fetch request from this consumer? Thanks, Jun On Tue, Feb 11,