Jason,

Thanks for sharing. This is very interesting. Normally, Kafka brokers don't
use too much CPU. Are most of the 750% CPU actually used by Kafka brokers?

Jun


On Wed, May 22, 2013 at 6:11 PM, Jason Weiss <jason_we...@rapid7.com> wrote:

> >>Did you check that you were using all cores?
>
> top was reporting over 750%
>
> Jason
>
> ________________________________________
> From: Ken Krugler [kkrugler_li...@transpac.com]
> Sent: Wednesday, May 22, 2013 20:59
> To: users@kafka.apache.org
> Subject: Re: Apache Kafka in AWS
>
> Hi Jason,
>
> On May 22, 2013, at 3:35pm, Jason Weiss wrote:
>
> > Ken,
> >
> > Great question! I should have indicated I was using EBS, 500GB with 2000
> provisioned IOPs.
>
> OK, thanks. Sounds like you were pegged on CPU usage.
>
> But that does surprise me a bit. Did you check that you were using all
> cores?
>
> Thanks,
>
> -- Ken
>
> PS - back in 2006 I spent a week of hell debugging an occasion job failure
> on Hadoop (this is when it was still part of Nutch). Turns out one of our
> 12 slaves was accidentally using OpenJDK, and this had a JIT compiler bug
> that would occasionally rear its ugly head. Obviously the Sun/Oracle JRE
> isn't bug-free, but it gets a lot more stress testing. So one of my basic
> guidelines in the ops portion of the Hadoop class I teach is that every
> server must have exactly the same version of Oracle's JRE.
>
> > ________________________________________
> > From: Ken Krugler [kkrugler_li...@transpac.com]
> > Sent: Wednesday, May 22, 2013 17:23
> > To: users@kafka.apache.org
> > Subject: Re: Apache Kafka in AWS
> >
> > Hi Jason,
> >
> > Thanks for the notes.
> >
> > I'm curious whether you went with using local drives (ephemeral storage)
> or EBS, and if with EBS then what IOPS.
> >
> > Thanks,
> >
> > -- Ken
> >
> > On May 22, 2013, at 1:42pm, Jason Weiss wrote:
> >
> >> All,
> >>
> >> I asked a number of questions of the group over the last week, and I'm
> happy to report that I've had great success getting Kafka up and running in
> AWS. I am using 3 EC2 instances, each of which is a M2 High-Memory
> Quadruple Extra Large with 8 cores and 58.4 GiB of memory according to the
> AWS specs. I have co-located Zookeeper instances next to Zafka on each
> machine.
> >>
> >> I am able to publish in a repeatable fashion 273,000 events per second,
> with each event payload consisting of a fixed size of 2048 bytes! This
> represents the maximum throughput possible on this configuration, as the
> servers became CPU constrained, averaging 97% utilization in a relatively
> flat line. This isn't a "burst" speed – it represents a sustained
> throughput from 20 M1 Large EC2 Kafka multi-threaded producers. Putting
> this into perspective, if my log retention period was a month, I'd be
> aggregating 1.3 petabytes of data on my disk drives. Suffice to say, I
> don't see us retaining data for more than a few hours!
> >>
> >> Here were the keys to tuning for future folks to consider:
> >>
> >> First and foremost, be sure to configure your Java heap size
> accordingly when you launch Kafka. The default is like 512MB, which in my
> case left virtually all of my RAM inaccessible to Kafka.
> >> Second, stay away from OpenJDK. No, seriously – this was a huge thorn
> in my side, and I almost gave up on Kafka because of the problems I
> encountered. The OpenJDK NIO functions repeatedly resulted in Kafka
> crashing and burning in dramatic fashion. The moment I switched over to
> Oracle's JDK for linux, Kafka didn't puke once- I mean, like not even a
> hiccup.
> >> Third know your message size. In my opinion, the more you understand
> about your event payload characteristics, the better you can tune the
> system. The two knobs to really turn are the log.flush.interval and
> log.default.flush.interval.ms. The values here are intrinsically
> connected to the types of payloads you are putting through the system.
> >> Fourth and finally, to maximize throughput you have to code against the
> async paradigm, and be prepared to tweak the batch size, queue properties,
> and compression codec (wait for it…) in a way that matches the message
> payload you are putting through the system and the capabilities of the
> producer system itself.
> >>
> >>
> >> Jason
> >>
> >>
> >>
> >>
> >>
> >> This electronic message contains information which may be confidential
> or privileged. The information is intended for the use of the individual or
> entity named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify us by e-mail at (
> postmas...@rapid7.com) immediately.
> >
> > --------------------------
> > Ken Krugler
> > +1 530-210-6378
> > http://www.scaleunlimited.com
> > custom big data solutions & training
> > Hadoop, Cascading, Cassandra & Solr
> >
> >
> >
> >
> >
> > This electronic message contains information which may be confidential
> or privileged. The information is intended for the use of the individual or
> entity named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify us by e-mail at (
> postmas...@rapid7.com) immediately.
> >
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>
>
> This electronic message contains information which may be confidential or
> privileged. The information is intended for the use of the individual or
> entity named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify us by e-mail at (
> postmas...@rapid7.com) immediately.
>
>

Reply via email to