I could not see such messages (Error while fetching metadata [%s] for topic [%s]: %s ")
In the perf test, I create new topic for sending the JSON blob in a new producer thread. The error happened when I increased no. of files (mapping to producer threads). There are already topics of previous performance test present in the kafka cluster having 0.8 head version. Regards, S.Abdullah On Wed, Oct 9, 2013 at 6:08 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Do you see WARN messages like - > > "Error while fetching metadata [%s] for topic [%s]: %s " > OR > "Error while fetching metadata %s for topic partition [%s,%d]: [%s]" > > > On Wed, Oct 9, 2013 at 5:49 PM, Shafaq <s.abdullah...@gmail.com> wrote: > > > yes I'm using 0.8 head > > > > > > roducer starting with option to multi-thread > > ----compressed---true > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException: Failed > > to fetch topic metadata for topic: 225topic1381362148396 > > at > > > > > kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53) > > at > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185) > > at > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149) > > at > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148) > > at > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57) > > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43) > > at > > > > > kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148) > > at > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94) > > at > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72) > > at kafka.producer.Producer.send(Producer.scala:74) > > at kafka.javaapi.producer.Producer.send(Producer.scala:32) > > at > > > > > gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:724) > > > > > > ----Processing file multithreaded > > mode----/mnt/prod_logs/relay_data_node-analytics-001_1380412084.gz > with > > compression enabled = true > > Exception in thread "pool-1-thread-2" kafka.common.KafkaException: Failed > > to fetch topic metadata for topic: 166topic1381362148385 > > at > > > > > kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53) > > at > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185) > > at > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149) > > at > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148) > > at > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57) > > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43) > > at > > > > > kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148) > > at > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94) > > at > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72) > > at kafka.producer.Producer.send(Producer.scala:74) > > at kafka.javaapi.producer.Producer.send(Producer.scala:32) > > at > > > > > gree.relayx.producer.distributed.GreeKafkaProducerDistributed.run(GreeKafkaProducerDistributed.java:124) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:724) > > > > > > On Wed, Oct 9, 2013 at 5:37 PM, Neha Narkhede <neha.narkh...@gmail.com > > >wrote: > > > > > Are you using 0.8 HEAD ? > > > > > > Can you send around the full stack trace since one of the common > reasons > > > for failed topic metadata requests are socket timeouts. > > > > > > Thanks, > > > Neha > > > > > > > > > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq <s.abdullah...@gmail.com> > wrote: > > > > > > > I run into following error while doing some performance testing in > > Kafka > > > > with producers running in multiple threads. > > > > > > > > I can see the topic under /broker/topics in zooKeeper > > > > > > > > I believe the producer tries to get the metadata info from Zookeeper. > > > > > > > > I have tried to restart the 2-node Kafka broker cluster but of no > use. > > > > > > > > Any idea what can cause this to happen? > > > > I'm using kafka 0.8 version > > > > > > > > > > > > > > --------------------------------------------------------------------------------------- > > > > > > > > Exception in thread "pool-1-thread-1" kafka.common.KafkaException: > > Failed > > > > to fetch topic metadata for topic: 37topic1381360774212 > > > > at > > > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53) > > > > at > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185) > > > > -- > > > > Kind Regards, > > > > Shafaq > > > > > > > > > > > > > > > -- > > Kind Regards, > > Shafaq > > > -- Kind Regards, Shafaq
