I'll try it and let you guys know. Is there anything that comes to mind from these log messages though? Why would there be so many log messages? Would you suggest doing something else to find out why things are working so poorly? I am worried about making the risky transition to the beta producer client and finding out that things don't improve at all. Another thing that was peculiar was that only one of my producers (out of 3, each on a separate host) were dropping so many messages.
On Sun, Dec 14, 2014 at 4:58 PM, Neha Narkhede <n...@confluent.io> wrote: > Thanks for reporting the issue, Rajiv. Since we are actively phasing out > the old client, it will be very helpful to know what the behavior on the > new client is. > > > On Fri, Dec 12, 2014 at 8:12 PM, Rajiv Kurian <ra...@signalfuse.com> > wrote: > > > > I am using the kafka java api async client (the one that wraps the Scala > > client). It's dropping quite a bit of data due to the queue being full, > and > > the process ends up taking a lot of cpu too. > > > > I am posting to a topic with 1024 partitions (across 3 brokers) - maybe > the > > high number of brokers is one of the issues. Profiling with YourKit > showed > > 37% of my CPU being spent on > kafka.producer.async.ProducerSendThread.run(). > > So seems like the producer is not able to keep up with my application and > > starts dropping. When I expand this waterfall in YourKit, I see that 23% > > total (not out of the 37%) is being spent on logging! Something like > this: > > > > > > > kafka.producer.BrokerPartitionInfo$$anonfun$getBrokerPartitionInfo$2.apply(PartitionMetadata) > > ->kafka.producer.BrokerPartitionInfo.debug(Function0) > > ->kafka.utils.Logging$class.debug(Logging, Function0) > > ->org.apache.log4j.Category.isDebugEnabled() > > ->... (a bunch of other things that finally break down into) > > ->LoggerContext.java:252 > > > > > ch.qos.logback.classic.spi.TurboFilterList.getTurboFilterChainDecision(Marker, > > Logger, Level, String, Object[], Throwable) > > > > I am not sure what's going on here. When I look at my process log, none > of > > these messages are actually logged (probably because of the log level). > > Further I don't see anything very suspicious on the broker logs. They are > > at 60-70% cpu. > > > > I am planning to try the new Java beta producer client, but I am afraid > > something deeper is going on here, that might not be solved by switching > to > > the newer client. > > > > > -- > Thanks, > Neha >
