With replication of 3, partitions that happen to be only hosted on those 3 shutdown brokers would be not available, unless controlled shutdown is used.
Guozhang On Sat, Aug 9, 2014 at 10:59 AM, Tanneru, Raj <[email protected]> wrote: > Guozhang, > > Our replication factor is 3. > > > Thanks, > Raj Tanneru > > > -------- Original message -------- > From: Guozhang Wang > Date:08/09/2014 10:21 AM (GMT-08:00) > To: [email protected] > Subject: Re: Error while producing messages > > Raj, > > What is your replication factor setting in kafka brokers? > > Guozhang > > > On Fri, Aug 8, 2014 at 5:02 PM, Tanneru, Raj <[email protected]> wrote: > > > Hi, > > > > > > > > I am trying to do capacity sizing estimate for our kafka cluster. I > > started with 5 broker cluster and 3 node zk. Used a simple java based > > producer to send messages to 5 topics that are created in the cluster. I > > used 2 client machines with 100 worker threads each sending messages > > continuously. I didn't see any exceptions or issues when all 5 brokers > are > > up. When I take down 3 brokers out of 5, and 4 topics out of 5. I am > seeing > > below in broker logs. > > > > > > > > [2014-08-08 16:47:34,977] DEBUG Closing connection from / > > 10.254.243.142:33944 (kafka.network.Processor) > > > > [2014-08-08 16:47:35,179] DEBUG Accepted connection from /10.254.243.142 > > on /10.66.107.231:9092. sendBufferSize [actual|requested]: > > [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] > > (kafka.network.Acceptor) > > > > [2014-08-08 16:47:35,179] DEBUG Processor 860 listening to new connection > > from /10.254.243.142:33947 (kafka.network.Processor) > > > > [2014-08-08 16:47:35,179] DEBUG [KafkaApi-1] Error while fetching > metadata > > for [item_topic_0,0]. Possible cause: null (kafka.server.KafkaApis) > > > > [2014-08-08 16:47:35,207] INFO Closing socket connection to / > > 10.254.243.142. (kafka.network.Processor) > > > > [2014-08-08 16:47:35,207] DEBUG Closing connection from / > > 10.254.243.142:33947 (kafka.network.Processor) > > > > [2014-08-08 16:47:35,294] DEBUG Accepted connection from /10.254.243.142 > > on /10.66.107.231:9092. sendBufferSize [actual|requested]: > > [131071|1048576] recvBufferSize [actual|requested]: [131071|1048576] > > (kafka.network.Acceptor) > > > > > > > > On the client machine(producer) I am seeing below error. Again only one > > topic and 2 broker nodes are running. > > > > > > > > Exception in thread "pool-1-thread-107" > > kafka.common.FailedToSendMessageException: Failed to send messages after > 3 > > tries. > > > > Sending message 181171284512 for topic item_topic_0 > > > > at > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) > > > > at kafka.producer.Producer.send(Producer.scala:76) > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33) > > > > at > > > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33) > > > > at > > > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > Exception in thread "pool-1-thread-108" > > kafka.common.FailedToSendMessageException: Failed to send messages after > 3 > > tries. > > > > Sending message 221252534837 for topic item_topic_0 > > > > at > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) > > > > at kafka.producer.Producer.send(Producer.scala:76) > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33) > > > > at > > > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.produceKafkaMessage(WorkerThread.java:33) > > > > at > > > com.ebay.cassini.feeder.nrt.kafka.producer.WorkerThread.run(WorkerThread.java:27) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > Not all messages are erroring out. Only a few messages are failing. Any > > idea what could be going on? > > > > > > > > Thanks, > > > > Raj Tanneru > > > > > > -- > -- Guozhang > -- -- Guozhang
