Re: QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

2015-06-19 Thread Mike Bridge
Ok, I've finally figured this out---it was a firewall/routing issue, but between zookeeper and the broker on the same machine, not between my client and the machine hosting zookeeper and the broker. Zookeeper uses the external IP address it receives from the advertised.host.name to connect back to

Re: QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

2015-06-17 Thread Mike Bridge
I set this up on EC2 in exactly the same way and had the same errors when accessing it with a producer that was outside EC2. Is there something else I have to configure other than to set "advertised.host.name" to my external IP address? On Tue, Jun 16, 2015 at 4:27 PM, Mike Bridge wrote: > Run

Re: QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

2015-06-16 Thread Mike Bridge
Running this seems to indicate that there is a leader at 0: $ ./bin/kafka-topics.sh --zookeeper MY.EXTERNAL.IP:2181 --describe --topic test123 --> Topic:test123 PartitionCount:1 ReplicationFactor:1 Configs: Topic: test123 Partition: 0 Leader: 0 Replicas: 0 Isr: 0 I reran this test and my

Re: QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

2015-06-16 Thread Gwen Shapira
The topic warning is a bug (i.e the fact that you get a warning on perfectly valid parameter). We fixed it for next release. It is also unrelated to the real issue with the LeaderNotAvailable On Tue, Jun 16, 2015 at 2:08 PM, Mike Bridge wrote: > I am able to get a simple one-node Kafka (kafka_2.

QuickStart OK locally, but getting "WARN Property topic is not valid" and LeaderNotAvailableException remotely

2015-06-16 Thread Mike Bridge
I am able to get a simple one-node Kafka (kafka_2.11-0.8.2.1) working locally on one linux machine, but when I try to run a producer remotely I'm getting some confusing errors. I'm following the quickstart guide at http://kafka.apache.org/documentation.html#quickstart. I stopped the kafka process

Re: Console Producer Throwing LeaderNotAvailableException Despite Existing Leader for Partition

2015-02-08 Thread Alex Melville
; > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) > > at > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) > > at > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) > > at > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > > at > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) > > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45) > > > > > > > > > > > > > > Clearly there's a leader, but it's throwing a > LeaderNotAvailableException. > > Any help here is appreciated. > > > > > > Cheers, > > > > Alex Melville > > > > > > -- > Regards, > Tao >

Re: Console Producer Throwing LeaderNotAvailableException Despite Existing Leader for Partition

2015-02-08 Thread tao xiao
dThread.scala:105) > at > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) > at > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > at > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) > at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45) > > > > > > > Clearly there's a leader, but it's throwing a LeaderNotAvailableException. > Any help here is appreciated. > > > Cheers, > > Alex Melville > -- Regards, Tao

Console Producer Throwing LeaderNotAvailableException Despite Existing Leader for Partition

2015-02-08 Thread Alex Melville
ucerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:526) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45) Clearly there's a leader, but it's throwing a LeaderNotAvailableEx

Re: Help: Kafka LeaderNotAvailableException

2015-01-22 Thread Gwen Shapira
It sounds like you have two zookeepers, one for HDP and one for Kafka. Did you move Kafka from one zookeeper to another? Perhaps Kafka finds the topics (logs) on disk, but they do not exist in ZK because you are using a different zookeeper now. Gwen On Thu, Jan 22, 2015 at 6:38 PM, Jun Rao wrot

Re: Help: Kafka LeaderNotAvailableException

2015-01-22 Thread Jun Rao
Any error in the controller and the broker log? Thanks, Jun On Thu, Jan 22, 2015 at 1:33 AM, wrote: > Hi, > Let me overview on the issue that I am facing on producing message in > Kafka: > I have horthonworks HDP-2.1 installed, along that we have Kafka on other > node. > > * On kafka node: > S

Re: Help: Kafka LeaderNotAvailableException

2015-01-22 Thread Joe Stein
Vishal, Does this error happen every time you are sending? Or just the first time? Joe Stein On Thu, Jan 22, 2015 at 4:33 AM, wrote: > Hi, > Let me overview on the issue that I am facing on producing message in > Kafka: > I have horthonworks HDP-2.1 installed, along that we have Kafka on other

Help: Kafka LeaderNotAvailableException

2015-01-22 Thread Vishal_Jain2
Hi, Let me overview on the issue that I am facing on producing message in Kafka: I have horthonworks HDP-2.1 installed, along that we have Kafka on other node. * On kafka node: Start Zookeepeer Start Kafka Broker service Send message/producer Consume message - Works (Note: here we start Zookeeper

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-15 Thread Abraham Jacob
Will definitely take a thread dump! So, far its been running fine. -Jacob On Wed, Oct 15, 2014 at 8:40 PM, Jun Rao wrote: > If you see the hanging again, it would be great if you can take a thread > dump so that we know where it is hanging. > > Thanks, > > Jun > > On Tue, Oct 14, 2014 at 10:35

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-15 Thread Jun Rao
If you see the hanging again, it would be great if you can take a thread dump so that we know where it is hanging. Thanks, Jun On Tue, Oct 14, 2014 at 10:35 PM, Abraham Jacob wrote: > Hi Jun, > > Thanks for responding... > > I am using Kafka 2.9.2-0.8.1.1 > > I looked through the controller lo

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-14 Thread Abraham Jacob
Hi Jun, Thanks for responding... I am using Kafka 2.9.2-0.8.1.1 I looked through the controller logs on a couple of nodes and did not find any exceptions or error. However in the state change log I see a bunch of the following exceptions - [2014-10-13 14:39:12,475] TRACE Controller 3 epoch 116

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-14 Thread Jun Rao
Also, which version of Kafka are you using? Thanks, Jun On Tue, Oct 14, 2014 at 5:31 PM, Jun Rao wrote: > The following is a bit weird. It indicates no leader for partition 4, > which is inconsistent with what describe-topic shows. > > 2014-10-13 19:02:32,611 WARN [main] kafka.producer.BrokerP

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-14 Thread Jun Rao
The following is a bit weird. It indicates no leader for partition 4, which is inconsistent with what describe-topic shows. 2014-10-13 19:02:32,611 WARN [main] kafka.producer.BrokerPartitionInfo: Error while fetching metadata partition 4 leader: nonereplicas: 3 (tr-pan-hclstr-13.amers1b.

Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-13 Thread Abraham Jacob
Hi All, I have a 8 node Kafka cluster (broker.id - 1..8). On this cluster I have a topic "wordcount", which was 8 partitions with a replication factor of 3. So a describe of topic wordcount # bin/kafka-topics.sh --describe --zookeeper tr-pan-hclstr-08.amers1b.ciscloud:2181/kafka/kafka-clstr-01 --

Re: LeaderNotAvailableException, although leader elected

2014-10-01 Thread Neha Narkhede
list.com/questions/4787268/kafka-common-leadernotavailableexception > ). > > For me this is isn't an acceptable way to communicate that the leader > election is in progress. There is not a single hint to this fact, but only > an exception. > If this is an expected behavior, the

Re: LeaderNotAvailableException, although leader elected

2014-09-27 Thread Andras Hatvani
ommon-leadernotavailableexception). For me this is isn't an acceptable way to communicate that the leader election is in progress. There is not a single hint to this fact, but only an exception. If this is an expected behavior, then it not only mustn't be an exception, but it

Re: LeaderNotAvailableException, although leader elected

2014-09-26 Thread Joel Koshy
> > kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], > > started (kafka.server.KafkaServer) > > kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) The above logs are for controller election. Not le

Re: LeaderNotAvailableException, although leader elected

2014-09-26 Thread Andras Hatvani
And the solution was: Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is: 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic) 11893 [main] INFO kafka.prod

LeaderNotAvailableException, although leader elected

2014-09-26 Thread Andras Hatvani
Hi, What does the exception mean in the following context? I've got two Kafka brokers out which the second one will be elected as leader: kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer) kafka2_1 | [2014-09-26 12:35:07,394] INFO Ne

Re: LeaderNotAvailableException

2014-08-13 Thread Ryan Williams
Ok, will do. Still with .8.1 on the same instance, after being reset has been running for about 48 hours now without a recurrence yet. On Wed, Aug 13, 2014 at 10:20 PM, Neha Narkhede wrote: > Due to KAFKA-1393, the server probably never ended up completely creating > the replicas. Let us know h

Re: LeaderNotAvailableException

2014-08-13 Thread Neha Narkhede
Due to KAFKA-1393, the server probably never ended up completely creating the replicas. Let us know how 0.8.1.1 goes. Thanks, Neha On Tue, Aug 12, 2014 at 10:12 AM, Ryan Williams wrote: > Using version 0.8.1. > > Looking to update to 0.8.1.1 now probably. > > > On Tue, Aug 12, 2014 at 9:25 AM,

Re: LeaderNotAvailableException

2014-08-12 Thread Ryan Williams
Using version 0.8.1. Looking to update to 0.8.1.1 now probably. On Tue, Aug 12, 2014 at 9:25 AM, Guozhang Wang wrote: > The "0" there in the kafka-topics output is the broker id. > > From the broker log I think you are hitting KAFKA-1393 > , wh

Re: LeaderNotAvailableException

2014-08-12 Thread Guozhang Wang
The "0" there in the kafka-topics output is the broker id. >From the broker log I think you are hitting KAFKA-1393 , which Kafka version are you using? Guozhang On Mon, Aug 11, 2014 at 10:37 PM, Ryan Williams wrote: > Thanks for the heads up o

Re: LeaderNotAvailableException

2014-08-11 Thread Ryan Williams
Thanks for the heads up on attachments, here's a gist: https://gist.githubusercontent.com/ryanwi/84deb8774a6922ff3704/raw/75c33ad71d0d41301533cbc645fa9846736d5eb0/gistfile1.txt This seems to mostly happen in my development environment, when running a single broker. I don't see any broker failure

Re: LeaderNotAvailableException

2014-08-11 Thread Ryan Williams
Thanks Neha, Indeed, there are no replicas apparently. $ bin/kafka-topics.sh --describe --zookeeper localhost:2181 Topic:eventsPartitionCount:2ReplicationFactor:1Configs: Topic: eventsPartition: 0Leader: 0Replicas: 0Isr: 0 Topic: eventsPartition: 1Leade

Re: LeaderNotAvailableException

2014-08-11 Thread Guozhang Wang
Ryan, Apache mailing list does not allow attachments exceeding a certain size limit, so the server logs is blocked. >From the controller log it seems this only broker has failed and hence no partitions will be available. This could be a soft failure (e.g. long GC), or the ZK server side issues. Y

Re: LeaderNotAvailableException

2014-08-11 Thread Neha Narkhede
[2014-08-11 19:12:45,321] ERROR Controller 0 epoch 3 initiated state change for partition [mytopic,0] from OfflinePartition to OnlinePartition failed (state.change.logger) kafka.common. NoReplicaOnlineException: No replica for partition [mytopic,0] is alive. Live brokers are: [Set()], Assigned repl

Re: LeaderNotAvailableException

2014-08-11 Thread Ryan Williams
The broker appears to be running $ telnet kafka-server 9092 Trying... Connected to kafka-server Escape character is '^]'. I've attached today's server.log. There was a manual restart of kafka, which you'll notice, but that didn't fix the issue. Thanks for looking! On Mon, Aug 11, 2014 a

Re: LeaderNotAvailableException

2014-08-11 Thread Guozhang Wang
Hi Ryan, Could you check if all of your brokers are still live and running? Also could you check the server log in addition to the producer / state-change / controller logs? Guozhang On Mon, Aug 11, 2014 at 12:45 PM, Ryan Williams wrote: > I have a single broker test Kafka instance that was r

LeaderNotAvailableException

2014-08-11 Thread Ryan Williams
I have a single broker test Kafka instance that was running fine on Friday (basically out of the box configuration with 2 partitions), now I come back on Monday and producers are unable to send messages. What else can i look at to debug, and prevent? I know how to recover by removing data directo

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-15 Thread Prakash Gowri Shankor
yes, I gave it several minutes. On Sat, Jun 14, 2014 at 2:18 PM, Michael G. Noll wrote: > Have you given Kafka some time to re-elect a new leader for the > "missing" partition when you re-try steps 1-5? > > See here: > > If you do, you should be able to go through steps > > 1-8 without seeing L

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-14 Thread Michael G. Noll
Have you given Kafka some time to re-elect a new leader for the "missing" partition when you re-try steps 1-5? See here: > If you do, you should be able to go through steps > 1-8 without seeing LeaderNotAvailableExceptions (you may need to give > Kafka some time to re-elect the remaining, second b

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-12 Thread Prakash Gowri Shankor
So if we go back to the 2 broker case, I tried your suggestion with replication-factor 2 ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor When i repeat steps 1-5 i still see the exception. When i go to step 8 ( back to 2 brokers ), I dont s

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-11 Thread Michael G. Noll
In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while. In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessio

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-11 Thread Prakash Gowri Shankor
Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/k

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-11 Thread Michael G. Noll
Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond "the original one". This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-11 Thread Prakash Gowri Shankor
; seem > > > to > > > > make it go away. The only thing that makes it go away is following > the > > > > steps listed in this stackoverflow thread. > > > > > > > > > > > > http://stackoverflow.com/questions/23228222/running-into-leadernotavailable

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-11 Thread Prakash Gowri Shankor
Is this what you want from kafka-topics ? I took this script dump now when the exception is occuring. ./kafka-topics.sh --describe test2 --zookeeper localhost:2181 Topic:test2 PartitionCount:3 ReplicationFactor:1 Configs: Topic: test2 Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: test2 Parti

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Jun Rao
Could you use the kafka-topic command to describe test2 and see if the leader is available? Thanks, Jun On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor < prakash.shan...@gmail.com> wrote: > Hi, > > I am running a cluster with a single broker, the performance producer > script and 3 con

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Joel Koshy
gt; On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote: > > > From the moment it starts occuring, it is persistent. Restarts dont seem > > to > > > make it go away. The only thing that makes it go away is following the > > > steps listed in this

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Guozhang Wang
the > > > steps listed in this stackoverflow thread. > > > > > > > > > http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3 > > > > > > > > > > > > On Tue, Jun 10, 2014 at 12:47

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Prakash Gowri Shankor
go away is following the > > steps listed in this stackoverflow thread. > > > > > http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3 > > > > > > > > On Tue, Jun 10, 2014 at 12:47

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Joel Koshy
22/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3 > > > > On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wrote: > > > Hello Prakash, > > > > Is this exception transient or persistent on broker startup? > > > > Gu

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Prakash Gowri Shankor
>From the moment it starts occuring, it is persistent. Restarts dont seem to make it go away. The only thing that makes it go away is following the steps listed in this stackoverflow thread. http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-

Re: LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Guozhang Wang
Hello Prakash, Is this exception transient or persistent on broker startup? Guozhang On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor < prakash.shan...@gmail.com> wrote: > Hi, > > I am running a cluster with a single broker, the performance producer > script and 3 consumers. > On a fres

LeaderNotAvailableException in 0.8.1.1

2014-06-10 Thread Prakash Gowri Shankor
Hi, I am running a cluster with a single broker, the performance producer script and 3 consumers. On a fresh start of the cluster , the producer throws this exception. I was able to run this cluster successfully on the same topic ( test2 ) successfully the first time. The solution( from stackover

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-29 Thread Rajasekar Elango
Created JIRA and attached patch to it. Please review. On Wed, Aug 28, 2013 at 1:11 PM, Guozhang Wang wrote: > I think this patch can be made in trunk. You can mark it as 0.8.1 > > Guozhang > > On Wednesday, August 28, 2013, Rajasekar Elango >

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-28 Thread Guozhang Wang
I think this patch can be made in trunk. You can mark it as 0.8.1 Guozhang On Wednesday, August 28, 2013, Rajasekar Elango wrote: > Guozhang , > > *The documentation says I need to work off of trunk. Can you confirm If I > should be working in trunk or different branch.* > * > * > *Thanks,* > *R

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-28 Thread Rajasekar Elango
t; > > > > > > > On Tue, Aug 27, 2013 at 9:45 AM, Guozhang Wang < > > wangg...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Rajasekar, > > > > > > &g

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-28 Thread Neha Narkhede
partition -> > > leader_broker_id > > > > map > > > > > > > which is used to determine to which brokers should the messages > > be > > > > > sent. > > > > > > > After new partitions are added, the cache on the produ

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-28 Thread Rajasekar Elango
> > > > > yet hence it will throw this exception. The producer will then > try > > to > > > > > > refresh its cache by asking the brokers "who are the leaders of > > these > > > > new > > > > > > partitions that I d

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Guozhang Wang
; > partitions that I do not know of before". The brokers at the > > beginning > > > > also > > > > > do not know this information, and will only get this information > from > > > > > controller which will only propagation the leader in

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Rajasekar Elango
> > > > > > > If you set num.retries to 3 then it is possible that producer gives > up > > > too > > > > soon before the leader info ever propagated to producers, hence to > > > > producers also. Could you try to increase producer.num.retries and > see

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Guozhang Wang
er propagated to producers, hence to > > > producers also. Could you try to increase producer.num.retries and see > if > > > the producer can eventually succeed in re-trying? > > > > > > Guozhang > > > > > > > > > On Tue, Aug 27,

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Rajasekar Elango
gt; > > On Tue, Aug 27, 2013 at 8:53 AM, Rajasekar Elango < > rela...@salesforce.com > > >wrote: > > > > > Hello everyone, > > > > > > We recently increased number of partitions from 4 to 16 and after that > > > console producer mostl

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Neha Narkhede
ceed in re-trying? > > Guozhang > > > On Tue, Aug 27, 2013 at 8:53 AM, Rajasekar Elango >wrote: > > > Hello everyone, > > > > We recently increased number of partitions from 4 to 16 and after that > > console producer mostly fails with LeaderNotAvailabl

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Guozhang Wang
ly succeed in re-trying? Guozhang On Tue, Aug 27, 2013 at 8:53 AM, Rajasekar Elango wrote: > Hello everyone, > > We recently increased number of partitions from 4 to 16 and after that > console producer mostly fails with LeaderNotAvailableException and exits > after 3 tries: &

Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

2013-08-27 Thread Rajasekar Elango
Hello everyone, We recently increased number of partitions from 4 to 16 and after that console producer mostly fails with LeaderNotAvailableException and exits after 3 tries: Here is last few lines of console producer log: No partition metadata for topic test-41 due to

Re: LeaderNotAvailableException

2013-08-07 Thread Vadim Keylis
Thanks so much sir. On Wed, Aug 7, 2013 at 8:00 AM, Jun Rao wrote: > For now, you can still add servers, but only newly created topics will go > there. If you just remove a server, you will be down 1 replica. What you > can do is to replace a server with a new one by keeping the same broker id.

Re: LeaderNotAvailableException

2013-08-07 Thread Jun Rao
For now, you can still add servers, but only newly created topics will go there. If you just remove a server, you will be down 1 replica. What you can do is to replace a server with a new one by keeping the same broker id. To recover from your error: (1) bring the 3 old brokers back up; (2) bring

Re: LeaderNotAvailableException

2013-08-07 Thread Vadim Keylis
Jun, What the process now if we want to add and remove servers? How can I recover from error in mean time? When is the final release? Thanks, Vadim On Wed, Aug 7, 2013 at 7:37 AM, Jun Rao wrote: > We do have a tool ReassignPartitionsCommand that allows you to move data > from one broker to an

Re: LeaderNotAvailableException

2013-08-07 Thread Jun Rao
We do have a tool ReassignPartitionsCommand that allows you to move data from one broker to another. It's still being tested and improved. It will be complete in the 0.8 final release. Thanks, Jun On Tue, Aug 6, 2013 at 11:18 PM, Vadim Keylis wrote: > Your assumption is correct. However I was

Re: LeaderNotAvailableException

2013-08-07 Thread Tejas Patil
Controlled shutdown transfers the leadership of the partitions from the broker (to be shutdown) to the anyone of the available in-sync replicas for that partition. It won't change the broker-partition assignments so that there would be atleast one live broker which would lead the partition. In you

Re: LeaderNotAvailableException

2013-08-06 Thread Vadim Keylis
Your assumption is correct. However I was under impression that shutdown should do rebalancing, was i wrong? What is the proper way to shutdown and allow partition to be serve by other brokers? What is the process to recover from this? How to avoid this and still have ability to remove servers f

Re: LeaderNotAvailableException

2013-08-06 Thread Tejas Patil
I assume that "We had 6 kafka in the tier" means that you had 6 kafka brokers. About the exception that you see: I think that the 3 brokers you took down were having the data for [junit2_analytics_data_log,0] and no other live broker has the data for [junit2_analytics_data_log,0]. You could run t

LeaderNotAvailableException

2013-08-06 Thread Vadim Keylis
We are using kafka08 beta1. We had 6 kafka in the tier. I have replication set to 3. Then 3 servers were removed using control shutdown method. I am getting the error bellow after that. What went wrong during shutdown? How to recover from the error? What steps to take in order to avoid in the futu

Re: LeaderNotAvailableException

2013-05-15 Thread Andrea Gazzarini
Hi, don't know exactly can help but I got the same LeaderNotAvailableException while playing (several restarts) with different versions of Zookeeper pointing to the same dataDir (without cleaning that between restarts) Regards, Andrea On 05/14/2013 06:12 PM, Yu, Libo wrote: Hi, I h

Re: LeaderNotAvailableException

2013-05-14 Thread Jun Rao
Any exceptions in the broker log (including state-change log)? Thanks, Jun On Tue, May 14, 2013 at 9:12 AM, Yu, Libo wrote: > Hi, > > I have 3 machines on a cluster. Normally, I start zookeepers on all three > before I launch kafka servers. Today for each machine I start zookeeper > first and

LeaderNotAvailableException

2013-05-14 Thread Yu, Libo
Hi, I have 3 machines on a cluster. Normally, I start zookeepers on all three before I launch kafka servers. Today for each machine I start zookeeper first and then kafka server. I noticed this error from one machine: The process was still running but it seems the server stopped responding. I pub

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-25 Thread Rob Withers
Yes, like in KAFKA-876's producer log, I only get one LeaderNotAvailableException. Thanks for fixing this issue, though could it only be fixed in the 0.8 branch? A colleague is having the issue tonight and I think he may be on the trunk. Like KAFKA-876, I am now getting the

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-25 Thread Jun Rao
Rob, Are you saying that LeaderNotAvailableException happens only once and then disappears? That is normal and is different from what happens in KAFKA-876. Thanks, Jun On Thu, Apr 25, 2013 at 8:41 PM, Withers, Robert wrote: > This LeaderNotAvailableException only occurs once now, runn

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-25 Thread Withers, Robert
This LeaderNotAvailableException only occurs once now, running kafka 8 from a dos shell. We are following KAFKA-876 closely which we also experience. thanks! rob From: Withers, Robert Sent: Friday, April 19, 2013 6:38 AM To: users@kafka.apache.org

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-21 Thread Jun Rao
s, unfortunately, running > kafka-console-consumer.bat and kafka-console-producer.bat. > > Best, > rob > > > From: Jun Rao [jun...@gmail.com] > Sent: Sunday, April 07, 2013 10:59 PM > To: users@kafka.apache.org > Subject: Re: LeaderNotAvailableException: reventing the run

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-19 Thread Withers, Robert
From: Jun Rao [jun...@gmail.com] Sent: Sunday, April 07, 2013 10:59 PM To: users@kafka.apache.org Subject: Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat That hash tag points to a commit on Apr 4 and is probably not what you have. Try "git log&qu

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-07 Thread Jun Rao
how I can get you the right info. > > Thanks, > Rob > > > > From: Jun Rao [jun...@gmail.com] > Sent: Friday, April 05, 2013 10:10 PM > To: users@kafka.apache.org > Subject: Re: LeaderNotAvailableException: reventing the running of >

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-06 Thread Withers, Robert
From: Jun Rao [jun...@gmail.com] Sent: Friday, April 05, 2013 10:10 PM To: users@kafka.apache.org Subject: Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat The following is the problem. The broker fails to become the leader. Do you know the

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-05 Thread Jun Rao
rT > hread) > [2013-04-05 10:32:08,137] INFO > [ConsumerFetcherThread-console-consumer-12815_MER > D7-178041-1365179527383-1da2aeba-0-0], Shutdown completed > (kafka.consumer.Consum > erFetcherThread) > [2013-04-05 10:32:08,153] INFO > [console-consumer-12815_MERD7-178041-1

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-05 Thread Shawn Hagwood
herThread-console-consumer-12815_MER > D7-178041-1365179527383-1da2aeba-0-0], Shutdown completed > (kafka.consumer.Consum > erFetcherThread) > [2013-04-05 10:32:08,153] INFO > [console-consumer-12815_MERD7-178041-136517952738 > 3-1da2aeba], ZKConsumerConnector shutting down >

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-05 Thread Withers, Robert
) [2013-04-05 10:32:08,568] INFO Session: 0x13ddafd0fc40002 closed (org.apache.zoo keeper.ZooKeeper) [2013-04-05 10:32:08,568] INFO EventThread shut down (org.apache.zookeeper.Clien tCnxn) Thanks, Rob -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, April 05, 20

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-05 Thread Jun Rao
and consumer console reported WARNs of > LeaderNotAvailableException. > > Since they were WARNs, would that mean that a message got produced and > then consumed? > > Thanks, > Rob > > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.co

RE: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-05 Thread Withers, Robert
and consumer console reported WARNs of LeaderNotAvailableException. Since they were WARNs, would that mean that a message got produced and then consumed? Thanks, Rob -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Thursday, April 04, 2013 10:03 PM To: users

Re: LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-04 Thread Jun Rao
producer.bat file and enter a "hello world" input to post to > the broker, the broker throws the LeaderNotAvailableException. > > Thanks for any assistance. > Rob > >

LeaderNotAvailableException: reventing the running of kafka-console-producer.bat

2013-04-04 Thread Withers, Robert
run the kafka-console-producer.bat file and enter a "hello world" input to post to the broker, the broker throws the LeaderNotAvailableException. Thanks for any assistance. Rob