AFAIK not topics, but only partitions of topics have leaders. What controllers do you mean? I haven't read about such.
Thanks for the explanation regarding the metadata request, in the meantime I found out that this is an expected (!) failure (http://qnalist.com/questions/4787268/kafka-common-leadernotavailableexception). For me this is isn't an acceptable way to communicate that the leader election is in progress. There is not a single hint to this fact, but only an exception. If this is an expected behavior, then it not only mustn't be an exception, but it also has to be communicated that there is something in progress. Furthermore, suggestions regarding changing the values variables I mentioned in my solution should be mandatory. This was my case: - OK, let's use Kafka - Create an infrastructure - Create a programmatic producer - Send a message - Message sending fails. - Retry - Message sending works! - Look for answers on the internet and in the docs - Read the configuration - Play around with configuration values. This is bad user experience especially for a newbie and involves a lot of effort. In my opinion programmatic message sending must work out of the box on the first try, without any exceptions, warnings or the need for additional configuration. I'd be glad to support/contribute. Regards, Andras > On 26 Sep 2014, at 19:53, Joel Koshy <jjkosh...@gmail.com> wrote: > >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], >>> started (kafka.server.KafkaServer) >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 >>> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > > The above logs are for controller election. Not leader election for > the topic you are producing to. > > When you producer starts it will issue a metadata request and it > should auto-create the topic (provided auto-create is on which is > default). The first metadata request for a non-existent topic always > returns LeaderNotAvailable because the controller then has to elect a > leader for the new topic. > > Joel > > On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote: >> And the solution was: >> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output >> is: >> >> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker >> id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) >> Set(inputTopic) >> 11893 [main] INFO kafka.producer.SyncProducer - Connected to >> 192.168.59.103:9092 for producing >> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from >> 192.168.59.103:9092 >> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching >> metadata [{TopicMetadata for topic inputTopic -> >> No partition metadata for topic inputTopic due to >> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class >> kafka.common.LeaderNotAvailableException >> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker >> id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s) >> Set(inputTopic) >> 12067 [main] INFO kafka.producer.SyncProducer - Connected to >> 192.168.59.103:9092 for producing >> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from >> 192.168.59.103:9092 >> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching >> metadata [{TopicMetadata for topic inputTopic -> >> No partition metadata for topic inputTopic due to >> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class >> kafka.common.LeaderNotAvailableException >> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to >> collate messages by topic, partition due to: Failed to fetch topic metadata >> for topic: inputTopic >> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for >> 1000 ms before retrying send. Remaining retries = 3 >> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker >> id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s) >> Set(inputTopic) >> 13111 [main] INFO kafka.producer.SyncProducer - Connected to >> 192.168.59.103:9092 for producing >> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from >> 192.168.59.103:9092 >> 13161 [main] INFO kafka.producer.SyncProducer - Connected to >> 192.168.59.103:9092 for producing >> >> In case this would be not enough for you, you can try to change the values >> of >> - message.send.max.retries from the default 5 to e.g. 10 and >> - topic.metadata.refresh.interval.ms to 0 >> >> >> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic >> metadata… >> >> >> Cheers, >> Andras >> >>> On 26 Sep 2014, at 14:43, Andras Hatvani <andras.hatv...@andrashatvani.com> >>> wrote: >>> >>> Hi, >>> >>> What does the exception mean in the following context? >>> >>> I've got two Kafka brokers out which the second one will be elected as >>> leader: >>> >>> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], >>> started (kafka.server.KafkaServer) >>> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 >>> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) >>> >>> Then, when I try to produce a message, I only get the following error: >>> >>> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for >>> 100 ms before retrying send. Remaining retries = 0 >>> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from >>> broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 >>> topic(s) Set(inputTopic) >>> 23287 [main] INFO kafka.producer.SyncProducer - Connected to >>> 192.168.59.103:9092 for producing >>> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from >>> 192.168.59.103:9092 >>> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while >>> fetching metadata [{TopicMetadata for topic inputTopic -> >>> No partition metadata for topic inputTopic due to >>> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class >>> kafka.common.LeaderNotAvailableException >>> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to >>> send requests for topics inputTopic with correlation ids in [0,8] >>> >>> The property metadata.broker.list contains the host:port values of both >>> brokers. >>> >>> What can be wrong/missing? >>> Thanks, >>> Andras >> >
