Hi Erik & Prabhjot

We are using Kafka-0.8.2.1 and old producer API with below config:

request.required.acks=1
request.timeout.ms=2000
producer.type=sync

On Kafka broker we are having:

num.network.threads=8
num.io.threads=10
num.replica.fetchers=4
replica.fetch.max.bytes=2097154
replica.fetch.wait.max.ms=500
replica.socket.timeout.ms=60000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000
replica.high.watermark.checkpoint.interval.ms=5000
replica.lag.max.messages=100

If you are asking about Singleton in terms of Producer then, we have
created pool of producers that has equal no of Producers and connection
that can be made in tomcat.


Thanks and Regards,
Madhukar

On Fri, Sep 11, 2015 at 8:27 PM, Prabhjot Bharaj <prabhbha...@gmail.com>
wrote:

> Hi,
>
> In addition to the parameters asked by Erik, it would be great if you could
> share your broker's server.properties as well
>
> Regards,
> Prabhjot
>
> On Fri, Sep 11, 2015 at 8:10 PM, Helleren, Erik <
> erik.helle...@cmegroup.com>
> wrote:
>
> > Hi Madhukar,
> > Some questions that can help understand whats going on: Which kafka
> > version is used?  Which Producer API is being used
> > (http://kafka.apache.org/documentation.html#producerapi)?  And what are
> > the configs for this producer?
> >
> > Also, because I know little about tomcat, is there a semantic for a
> > singleton, or a server singleton?
> > -Erik
> >
> > On 9/11/15, 8:48 AM, "Madhukar Bharti" <bhartimadhu...@gmail.com> wrote:
> >
> > >Hi,
> > >
> > >
> > >We are having 3 brokers in a cluster. Producer request is getting failed
> > >for broker 2. We are frequently getting below exception:
> > >
> > >15/09/09 22:09:06 WARN async.DefaultEventHandler: Failed to send
> > >producer request with* correlation id 1455 to broker 2* with data for
> > >partitions [UserEvents,0]
> > >> java.net.SocketTimeoutException
> > >>      at
> > >>sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> > >>      at
> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> > >>      at
> >
> >>java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385
> > >>)
> > >>      at kafka.utils.Utils$.read(Utils.scala:375)
> > >>      at
> >
> >>kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.
> > >>scala:54)
> > >>      at
> > kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > >>      at
> >
> >>kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferRe
> > >>ceive.scala:29)
> > >>      at
> kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > >>
> > >>
> > >After looking into request-logs in all machines, found that there is
> some
> > >slowness in broker 2. I am listing top 20 request processing time from
> all
> > >the brokers.
> > >
> > >Broker 1
> > >
> > >  Broker 2
> > >
> > >                       Broker 3
> > >
> > >Producer&
> > >
> > >Fetcher
> > >
> > >Producer
> > >
> > >Producer + Fetcher
> > >
> > >Producer
> > >
> > >            Producer +  Fetcher
> > >
> > >Producer
> > >
> > >493
> > >
> > >494
> > >
> > >495
> > >
> > >496
> > >
> > >497
> > >
> > >498
> > >
> > >499
> > >
> > >500
> > >
> > >501
> > >
> > >502
> > >
> > >503
> > >
> > >504
> > >
> > >519
> > >
> > >520
> > >
> > >541
> > >
> > >542
> > >
> > >545
> > >
> > >551
> > >
> > >577
> > >
> > >633
> > >
> > >77
> > >
> > >91
> > >
> > >94
> > >
> > >96
> > >
> > >104
> > >
> > >111
> > >
> > >112
> > >
> > >153
> > >
> > >167
> > >
> > >184
> > >
> > >248
> > >
> > >249
> > >
> > >254
> > >
> > >284
> > >
> > >395
> > >
> > >443
> > >
> > >470
> > >
> > >551
> > >
> > >577
> > >
> > >633
> > >
> > >1033
> > >
> > >1034
> > >
> > >1035
> > >
> > >1036
> > >
> > >1037
> > >
> > >1038
> > >
> > >1039
> > >
> > >1040
> > >
> > >1042
> > >
> > >1043
> > >
> > >1044
> > >
> > >1049
> > >
> > >1051
> > >
> > >1057
> > >
> > >1064
> > >
> > >1087
> > >
> > >1145
> > >
> > >1146
> > >
> > >1466
> > >
> > >1467
> > >
> > >85
> > >
> > >86
> > >
> > >114
> > >
> > >121
> > >
> > >123
> > >
> > >136
> > >
> > >153
> > >
> > >201
> > >
> > >225
> > >
> > >226
> > >
> > >240
> > >
> > >299
> > >
> > >405
> > >
> > >406
> > >
> > >448
> > >
> > >449
> > >
> > >455
> > >
> > >464
> > >
> > >505
> > >
> > >658
> > >
> > >489
> > >
> > >490
> > >
> > >491
> > >
> > >492
> > >
> > >493
> > >
> > >494
> > >
> > >495
> > >
> > >496
> > >
> > >497
> > >
> > >498
> > >
> > >499
> > >
> > >500
> > >
> > >501
> > >
> > >502
> > >
> > >503
> > >
> > >506
> > >
> > >510
> > >
> > >514
> > >
> > >515
> > >
> > >516
> > >
> > >19
> > >
> > >20
> > >
> > >21
> > >
> > >22
> > >
> > >23
> > >
> > >24
> > >
> > >27
> > >
> > >28
> > >
> > >31
> > >
> > >32
> > >
> > >60
> > >
> > >89
> > >
> > >98
> > >
> > >104
> > >
> > >110
> > >
> > >114
> > >
> > >259
> > >
> > >288
> > >
> > >337
> > >
> > >385
> > >
> > >
> > >What can be the reason that fetcher thread taking more time to process?
> > >
> > >What we need to do to get better performance? Any properties we need to
> > >tweak?
> > >
> > >Any suggestion are welcome.
> > >
> > >
> > >Note: We are pushing data to Kafka in user thread(tomcat) and set
> producer
> > >request timeout to 2sec. We don't want to increase timeout more than 2
> > >sec., as if it too many threads will get hangup then application will be
> > >hanged.
> > >
> > >
> > >Thanks and Regards,
> > >Madhukar
> >
> >
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>

Reply via email to