Hi,
We are having 3 brokers in a cluster. Producer request is getting failed for broker 2. We are frequently getting below exception: 15/09/09 22:09:06 WARN async.DefaultEventHandler: Failed to send producer request with* correlation id 1455 to broker 2* with data for partitions [UserEvents,0] > java.net.SocketTimeoutException > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > at kafka.utils.Utils$.read(Utils.scala:375) > at > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > at kafka.network.Receive$class.readCompletely(Transmission.scala:56) > at > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) > > After looking into request-logs in all machines, found that there is some slowness in broker 2. I am listing top 20 request processing time from all the brokers. Broker 1 Broker 2 Broker 3 Producer& Fetcher Producer Producer + Fetcher Producer Producer + Fetcher Producer 493 494 495 496 497 498 499 500 501 502 503 504 519 520 541 542 545 551 577 633 77 91 94 96 104 111 112 153 167 184 248 249 254 284 395 443 470 551 577 633 1033 1034 1035 1036 1037 1038 1039 1040 1042 1043 1044 1049 1051 1057 1064 1087 1145 1146 1466 1467 85 86 114 121 123 136 153 201 225 226 240 299 405 406 448 449 455 464 505 658 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 506 510 514 515 516 19 20 21 22 23 24 27 28 31 32 60 89 98 104 110 114 259 288 337 385 What can be the reason that fetcher thread taking more time to process? What we need to do to get better performance? Any properties we need to tweak? Any suggestion are welcome. Note: We are pushing data to Kafka in user thread(tomcat) and set producer request timeout to 2sec. We don't want to increase timeout more than 2 sec., as if it too many threads will get hangup then application will be hanged. Thanks and Regards, Madhukar