RE: Apache Kafka in AWS

2013-05-29 Thread Jason Weiss
Nope, it was specific to the application I'm working on carrying real-world data. Sorry :-( Jason From: S Ahmed [sahmed1...@gmail.com] Sent: Wednesday, May 29, 2013 13:40 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Is the code you

Re: Apache Kafka in AWS

2013-05-29 Thread S Ahmed
> > >>screenshot of > > >> the top output. > > >> > > >> This shows when I added a 4th machine to the cluster with the same > > >>number > > >> of clients, my CPU utilization fell- but remained constant. The > > >>fl

RE: Apache Kafka in AWS

2013-05-28 Thread Jason Weiss
Nope, sorry. From: S Ahmed [sahmed1...@gmail.com] Sent: Tuesday, May 28, 2013 15:47 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Curious if you tested with larger message sizes, like around 20-30kb (you mentioned 2kb). Any numbers on that

Re: Apache Kafka in AWS

2013-05-28 Thread S Ahmed
onstant. The > >>flatline is > >> pretty obvious in the extended 4 minute test-- it ramps up, flat lines, > >> then ramps down. > >> > >> Jason > >> > >> > >> From: Jun Rao [jun...@gmail.com]

Re: Apache Kafka in AWS

2013-05-23 Thread Jason Weiss
ed constant. The >>flatline is >> pretty obvious in the extended 4 minute test-- it ramps up, flat lines, >> then ramps down. >> >> Jason >> >> ____ >> From: Jun Rao [jun...@gmail.com] >> Sent: Thursday, May 23, 2013 00:17 >> To: users@kafka.apache.or

Re: Apache Kafka in AWS

2013-05-23 Thread Jun Rao
ine is > pretty obvious in the extended 4 minute test-- it ramps up, flat lines, > then ramps down. > > Jason > > > From: Jun Rao [jun...@gmail.com] > Sent: Thursday, May 23, 2013 00:17 > To: users@kafka.apache.org > Subject: Re

RE: Apache Kafka in AWS

2013-05-23 Thread Jason Weiss
t remained constant. The flatline is pretty obvious in the extended 4 minute test-- it ramps up, flat lines, then ramps down. Jason From: Jun Rao [jun...@gmail.com] Sent: Thursday, May 23, 2013 00:17 To: users@kafka.apache.org Subject: Re: Apache Kafka in

Re: Apache Kafka in AWS

2013-05-22 Thread Jun Rao
p was reporting over 750% > > Jason > > > From: Ken Krugler [kkrugler_li...@transpac.com] > Sent: Wednesday, May 22, 2013 20:59 > To: users@kafka.apache.org > Subject: Re: Apache Kafka in AWS > > Hi Jason, > > On May 22, 2013

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
>>Did you check that you were using all cores? top was reporting over 750% Jason From: Ken Krugler [kkrugler_li...@transpac.com] Sent: Wednesday, May 22, 2013 20:59 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi Jason, On May 22

Re: Apache Kafka in AWS

2013-05-22 Thread Ken Krugler
_ > From: Ken Krugler [kkrugler_li...@transpac.com] > Sent: Wednesday, May 22, 2013 17:23 > To: users@kafka.apache.org > Subject: Re: Apache Kafka in AWS > > Hi Jason, > > Thanks for the notes. > > I'm curious whether you went with using local drives (ephemeral sto

Re: Apache Kafka in AWS

2013-05-22 Thread Scott Clasen
t; > Jason > > > > From: Scott Clasen [sc...@heroku.com] > Sent: Wednesday, May 22, 2013 19:27 > To: users > Subject: Re: Apache Kafka in AWS > > Hey Jason, > > question what openjdk version did you have issues with? Im

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
cott Clasen [sc...@heroku.com] Sent: Wednesday, May 22, 2013 19:27 To: users Subject: Re: Apache Kafka in AWS Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013 at 1:42 PM, Jas

Re: Apache Kafka in AWS

2013-05-22 Thread Scott Clasen
Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013 at 1:42 PM, Jason Weiss wrote: > All, > > I asked a number of questions of the group over the last week, and I'm > happy

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
22, 2013 19:11 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Awesome right up Jason! Very helpful as we are also looking to build a Kafka environment in AWS. I am curious, are you using Kafka 0.7.2 or 0.8 in your tests? Did you have just one EBS volume per broker instance or RAID 10 acro

Re: Apache Kafka in AWS

2013-05-22 Thread Jonathan Hodges
Sent: Wednesday, May 22, 2013 17:23 > To: users@kafka.apache.org > Subject: Re: Apache Kafka in AWS > > Hi Jason, > > Thanks for the notes. > > I'm curious whether you went with using local drives (ephemeral storage) > or EBS, and if with EBS then what IOPS. >

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
Ken, Great question! I should have indicated I was using EBS, 500GB with 2000 provisioned IOPs. Jason From: Ken Krugler [kkrugler_li...@transpac.com] Sent: Wednesday, May 22, 2013 17:23 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi

Re: Apache Kafka in AWS

2013-05-22 Thread Ken Krugler
Hi Jason, Thanks for the notes. I'm curious whether you went with using local drives (ephemeral storage) or EBS, and if with EBS then what IOPS. Thanks, -- Ken On May 22, 2013, at 1:42pm, Jason Weiss wrote: > All, > > I asked a number of questions of the group over the last week, and I'm ha

Re: Apache Kafka in AWS

2013-05-22 Thread Neha Narkhede
Thanks for sharing your experience with the community, Jason! -Neha On Wed, May 22, 2013 at 1:42 PM, Jason Weiss wrote: > All, > > I asked a number of questions of the group over the last week, and I'm > happy to report that I've had great success getting Kafka up and running in > AWS. I am us