Hi, In addition to the parameters asked by Erik, it would be great if you could share your broker's server.properties as well
Regards, Prabhjot On Fri, Sep 11, 2015 at 8:10 PM, Helleren, Erik <erik.helle...@cmegroup.com> wrote: > Hi Madhukar, > Some questions that can help understand whats going on: Which kafka > version is used? Which Producer API is being used > (http://kafka.apache.org/documentation.html#producerapi)? And what are > the configs for this producer? > > Also, because I know little about tomcat, is there a semantic for a > singleton, or a server singleton? > -Erik > > On 9/11/15, 8:48 AM, "Madhukar Bharti" <bhartimadhu...@gmail.com> wrote: > > >Hi, > > > > > >We are having 3 brokers in a cluster. Producer request is getting failed > >for broker 2. We are frequently getting below exception: > > > >15/09/09 22:09:06 WARN async.DefaultEventHandler: Failed to send > >producer request with* correlation id 1455 to broker 2* with data for > >partitions [UserEvents,0] > >> java.net.SocketTimeoutException > >> at > >>sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) > >> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > >> at > >>java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385 > >>) > >> at kafka.utils.Utils$.read(Utils.scala:375) > >> at > >>kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive. > >>scala:54) > >> at > kafka.network.Receive$class.readCompletely(Transmission.scala:56) > >> at > >>kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferRe > >>ceive.scala:29) > >> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) > >> > >> > >After looking into request-logs in all machines, found that there is some > >slowness in broker 2. I am listing top 20 request processing time from all > >the brokers. > > > >Broker 1 > > > > Broker 2 > > > > Broker 3 > > > >Producer& > > > >Fetcher > > > >Producer > > > >Producer + Fetcher > > > >Producer > > > > Producer + Fetcher > > > >Producer > > > >493 > > > >494 > > > >495 > > > >496 > > > >497 > > > >498 > > > >499 > > > >500 > > > >501 > > > >502 > > > >503 > > > >504 > > > >519 > > > >520 > > > >541 > > > >542 > > > >545 > > > >551 > > > >577 > > > >633 > > > >77 > > > >91 > > > >94 > > > >96 > > > >104 > > > >111 > > > >112 > > > >153 > > > >167 > > > >184 > > > >248 > > > >249 > > > >254 > > > >284 > > > >395 > > > >443 > > > >470 > > > >551 > > > >577 > > > >633 > > > >1033 > > > >1034 > > > >1035 > > > >1036 > > > >1037 > > > >1038 > > > >1039 > > > >1040 > > > >1042 > > > >1043 > > > >1044 > > > >1049 > > > >1051 > > > >1057 > > > >1064 > > > >1087 > > > >1145 > > > >1146 > > > >1466 > > > >1467 > > > >85 > > > >86 > > > >114 > > > >121 > > > >123 > > > >136 > > > >153 > > > >201 > > > >225 > > > >226 > > > >240 > > > >299 > > > >405 > > > >406 > > > >448 > > > >449 > > > >455 > > > >464 > > > >505 > > > >658 > > > >489 > > > >490 > > > >491 > > > >492 > > > >493 > > > >494 > > > >495 > > > >496 > > > >497 > > > >498 > > > >499 > > > >500 > > > >501 > > > >502 > > > >503 > > > >506 > > > >510 > > > >514 > > > >515 > > > >516 > > > >19 > > > >20 > > > >21 > > > >22 > > > >23 > > > >24 > > > >27 > > > >28 > > > >31 > > > >32 > > > >60 > > > >89 > > > >98 > > > >104 > > > >110 > > > >114 > > > >259 > > > >288 > > > >337 > > > >385 > > > > > >What can be the reason that fetcher thread taking more time to process? > > > >What we need to do to get better performance? Any properties we need to > >tweak? > > > >Any suggestion are welcome. > > > > > >Note: We are pushing data to Kafka in user thread(tomcat) and set producer > >request timeout to 2sec. We don't want to increase timeout more than 2 > >sec., as if it too many threads will get hangup then application will be > >hanged. > > > > > >Thanks and Regards, > >Madhukar > > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
