Thanks Scott & Monika. I just followed the Kafka documentation to start the zookeeper & brokers. They did not mention of this zookeeper id maintenance.
One more observation is, it works fine after several retries. Sometimes the zookeepers could not recognize the brokers etc. -----Original Message----- From: Monika Garg [mailto:gargmon...@gmail.com] Sent: Saturday, August 8, 2015 6:52 AM To: users@kafka.apache.org Subject: Re: Inconsistency with Zookeeper Please check and ensure that each machine can recognize each other. It might be the case zookeepers and kafka are running seperately on each machine, but together they are not part of the same cluster. On 08-Aug-2015 6:34 am, "Scott Clasen" <sc...@heroku.com> wrote: > each zk needs a myid file in the data dir, with a different number > 1,2,3 > > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html > > You can find the meanings of these and other configuration settings in > the section Configuration Parameters < > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_configur > ation > >. > A word though about a few here: > > Every machine that is part of the ZooKeeper ensemble should know about > every other machine in the ensemble. You accomplish this with the > series of lines of the form*server.id > <http://server.id>=host:port:port*. The parameters *host* and *port* > are straightforward. You attribute the server id to each machine by > creating a file named myid, one for each server, which resides in that > server's data directory, as specified by the configuration file parameter > *dataDir*. > > On Fri, Aug 7, 2015 at 5:59 PM, Hemanth Abbina > <heman...@eiqnetworks.com> > wrote: > > > Yes. I have set the unique broker ids (as 0, 1, 2) in the > > server.properties file > > > > I did not get the " set broker id in zookeeper data directory". We > don't > > set any broker id in zookeeper. > > > > We provide only "zookeeper.connect=ZK:2181,ZK2:2181,ZK3:2181" in > > server.properties file. Right ? > > > > -----Original Message----- > > From: Prabhjot Bharaj [mailto:prabhbha...@gmail.com] > > Sent: Friday, August 7, 2015 8:44 PM > > To: users@kafka.apache.org > > Subject: Re: Inconsistency with Zookeeper > > > > Have you set broker id in zookeeper data directory and some unique > broker > > Id in server.properties ? > > > > Regards, > > Prabhjot > > On Aug 6, 2015 1:43 PM, "Hemanth Abbina" <heman...@eiqnetworks.com> > wrote: > > > > > Hi, > > > > > > I am running a Kafka POC with below details > > > > > > * 3 Node cluster (4 Core, 16 GB RAM each) running Kafka > 0.8.2.1. > > > > > > * Each node running Kafka & Zookeeper instances. (So total of 3 > > > Kafka brokers & 3 zookeepers) > > > > > > When I tried to create a topic using the kafka-topics.sh, > > > observing the below exceptions on the console. If I try multiple > > > times, the topic is getting created at some time. Please suggest a > > > workaround to make this work every time. > > > > > > Thanks in advance. > > > > > > [ec2-user@ip-10-20-0-196 kafka_2.10-0.8.2.1]$ > > > ./bin/kafka-topics.sh --zookeeper > > > 10.20.0.196:2181,10.20.0.197:2181,10.20.0.198:2181 --topic > > > reportspoc4 --create --partitions 1 --replication-factor 2 Error > > > while executing topic command replication factor: 2 larger than > > > available > > > brokers: 1 > > > kafka.admin.AdminOperationException: replication factor: 2 larger > > > than available brokers: 1 > > > at > > > kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70) > > > at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171) > > > at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:93) > > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:55) > > > at kafka.admin.TopicCommand.main(TopicCommand.scala) > > > > > > > > > [ec2-user@ip-10-20-0-196 kafka_2.10-0.8.2.1]$ > > > ./bin/kafka-topics.sh --zookeeper > > > 10.20.0.196:2181,10.20.0.197:2181,10.20.0.198:2181 --topic > > > reportspoc4 --create --partitions 1 --replication-factor 2 Error > > > while executing topic command > > > org.apache.zookeeper.KeeperException$NoNodeException: > > > KeeperErrorCode = NoNode for /brokers/ids > > > org.I0Itec.zkclient.exception.ZkNoNodeException: > > > org.apache.zookeeper.KeeperException$NoNodeException: > > > KeeperErrorCode = NoNode for /brokers/ids > > > at > > > org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) > > > at > > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) > > > at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) > > > at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) > > > at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:462) > > > at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:78) > > > at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:170) > > > at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:93) > > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:55) > > > at kafka.admin.TopicCommand.main(TopicCommand.scala) > > > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > > > KeeperErrorCode = NoNode for /brokers/ids > > > at > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > > > at > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > > at > > org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) > > > at > > org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) > > > at > > > org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) > > > at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) > > > at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) > > > at > > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) > > > ... 8 more > > > > > > --regards > > > Hemanth > > > > > >
