Jun,

I re-ran the hprof test, for about 30 minutes again, for 0.8.2.0-rc2 with
the same version of snappy that 0.8.1.1 used. Attached the logs.
Unfortunately there wasn't any improvement as the node running 0.8.2.0-rc2
still had a higher load and CPU usage.

Best regards,
Mathias

On Tue Feb 03 2015 at 4:40:31 AM Jaikiran Pai <jai.forums2...@gmail.com>
wrote:

> On Monday 02 February 2015 11:03 PM, Jun Rao wrote:
> > Jaikiran,
> >
> > The fix you provided in probably unnecessary. The channel that we use in
> > SimpleConsumer (BlockingChannel) is configured to be blocking. So even
> > though the read from the socket is in a loop, each read blocks if there
> is
> > no bytes received from the broker. So, that shouldn't cause extra CPU
> > consumption.
> Hi Jun,
>
> Of course, you are right! I forgot that while reading the thread dump in
> hprof output, one has to be aware that the thread state isn't shown and
> the thread need not necessarily be doing any CPU activity.
>
> -Jaikiran
>
>
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 10:05 AM, Mathias Söderberg <
> > mathias.soederb...@gmail.com> wrote:
> >
> >> Hi Neha,
> >>
> >> I sent an e-mail earlier today, but noticed now that it didn't actually
> go
> >> through.
> >>
> >> Anyhow, I've attached two files, one with output from a 10 minute run
> and
> >> one with output from a 30 minute run. Realized that maybe I should've
> done
> >> one or two runs with 0.8.1.1 as well, but nevertheless.
> >>
> >> I upgraded our staging cluster to 0.8.2.0-rc2, and I'm seeing the same
> CPU
> >> usage as with the beta version (basically pegging all cores). If I
> manage
> >> to find the time I'll do another run with hprof on the rc2 version later
> >> today.
> >>
> >> Best regards,
> >> Mathias
> >>
> >> On Tue Dec 09 2014 at 10:08:21 PM Neha Narkhede <n...@confluent.io>
> wrote:
> >>
> >>> The following should be sufficient
> >>>
> >>> java
> >>> -agentlib:hprof=cpu=samples,depth=100,interval=20,lineno=
> >>> y,thread=y,file=kafka.hprof
> >>> <classname>
> >>>
> >>> You would need to start the Kafka server with the settings above for
> >>> sometime until you observe the problem.
> >>>
> >>> On Tue, Dec 9, 2014 at 3:47 AM, Mathias Söderberg <
> >>> mathias.soederb...@gmail.com> wrote:
> >>>
> >>>> Hi Neha,
> >>>>
> >>>> Yeah sure. I'm not familiar with hprof, so any particular options I
> >>> should
> >>>> include or just run with defaults?
> >>>>
> >>>> Best regards,
> >>>> Mathias
> >>>>
> >>>> On Mon Dec 08 2014 at 7:41:32 PM Neha Narkhede <n...@confluent.io>
> >>> wrote:
> >>>>> Thanks for reporting the issue. Would you mind running hprof and
> >>> sending
> >>>>> the output?
> >>>>>
> >>>>> On Mon, Dec 8, 2014 at 1:25 AM, Mathias Söderberg <
> >>>>> mathias.soederb...@gmail.com> wrote:
> >>>>>
> >>>>>> Good day,
> >>>>>>
> >>>>>> I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed
> >>>> that
> >>>>>> the CPU usage on the broker machines went up by roughly 40%, from
> >>> ~60%
> >>>> to
> >>>>>> ~100% and am wondering if anyone else has experienced something
> >>>> similar?
> >>>>>> The load average also went up by 2x-3x.
> >>>>>>
> >>>>>> We're running on EC2 and the cluster currently consists of four
> >>>>> m1.xlarge,
> >>>>>> with roughly 1100 topics / 4000 partitions. Using Java 7 (1.7.0_65
> >>> to
> >>>> be
> >>>>>> exact) and Scala 2.9.2. Configurations can be found over here:
> >>>>>> https://gist.github.com/mthssdrbrg/7df34a795e07eef10262.
> >>>>>>
> >>>>>> I'm assuming that this is not expected behaviour for 0.8.2-beta?
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Mathias
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Thanks,
> >>>>> Neha
> >>>>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Neha
> >>>
>
>

Attachment: kafka-0.8.2.0-snappy-20150203.hprof.gz
Description: GNU Zip compressed data

Reply via email to