Re: Kafka on yarn

2014-07-24 Thread Kam Kasravi
;s no coincidence that as YARN evolves the difference between YARN and a cluster manager like ambari shrink. On Wednesday, July 23, 2014 6:41 PM, Gwen Shapira wrote: Hi, Can we discuss for a moment the use-case of Kafka-on-YARN? I (as Cloudera field engineer) typically advise my customers to i

Re: Kafka on yarn

2014-07-24 Thread Kam Kasravi
th replication you can restart a > broker elsewhere with no data, and it will restore it's state off the > other replicas. > > -Jay > > > On Wed, Jul 23, 2014 at 3:47 PM, Kam Kasravi > wrote: >> Hi >> >> Kafka-on-yarn requires YARN to consistently al

Re: Kafka on yarn

2014-07-24 Thread Kam Kasravi
to date.  On Wednesday, July 23, 2014 5:15 PM, Steve Morin wrote: Kam,   Give it some time and think it's getting better as a real possibility for Kafka on Yarn.  There are new capabilities coming out in Yarn/HDFS to allow for node groups/label that can work with locality

Re: Kafka on yarn

2014-07-23 Thread Gwen Shapira
Hi, Can we discuss for a moment the use-case of Kafka-on-YARN? I (as Cloudera field engineer) typically advise my customers to install Kafka on their own nodes, to allow Kafka uninterrupted access to disks. Hadoop processes tend to be a bit IO heavy. Also, I can't see any benefit fr

Re: Kafka on yarn

2014-07-23 Thread hsy...@gmail.com
ion. As long as you run with replication you can restart a > broker elsewhere with no data, and it will restore it's state off the > other replicas. > > -Jay > > > On Wed, Jul 23, 2014 at 3:47 PM, Kam Kasravi > wrote: > > Hi > > > > Kafka-on-yarn requ

Re: Kafka on yarn

2014-07-23 Thread Jay Kreps
t; It would be nice to have a way to get a failed node back with it's > original data, but this isn't strictly necessary, it is just a good > optimization. As long as you run with replication you can restart a > broker elsewhere with no data, and it will restore it's state of

Re: Kafka on yarn

2014-07-23 Thread Steve Morin
Kam, Give it some time and think it's getting better as a real possibility for Kafka on Yarn. There are new capabilities coming out in Yarn/HDFS to allow for node groups/label that can work with locality and secondarily new functionality in HDFS that depending on the use-case can be

Re: Kafka on yarn

2014-07-23 Thread Kam Kasravi
inal data, but this isn't strictly necessary, it is just a good optimization. As long as you run with replication you can restart a broker elsewhere with no data, and it will restore it's state off the other replicas. -Jay On Wed, Jul 23, 2014 at 3:47 PM, Kam Kasravi wrote: >

Re: Kafka on yarn

2014-07-23 Thread Jay Kreps
other replicas. -Jay On Wed, Jul 23, 2014 at 3:47 PM, Kam Kasravi wrote: > Hi > > Kafka-on-yarn requires YARN to consistently allocate a kafka broker at a > particular resource since the broker needs to always use its local data. YARN > doesn't do this well, unless you provi

Re: Kafka on yarn

2014-07-23 Thread Joe Stein
y...@gmail.com wrote: > Hi guys, > > Kafka is getting more and more popular and in most cases people run kafka > as long-term service in the cluster. Is there a discussion of running kafka > on yarn cluster which we can utilize the convenient configuration/resource > management and HA

Re: Kafka on yarn

2014-07-23 Thread Kam Kasravi
Hi  Kafka-on-yarn requires YARN to consistently allocate a kafka broker at a particular resource since the broker needs to always use its local data. YARN doesn't do this well, unless you provide (override) the default scheduler (CapacityScheduler or FairScheduler). SequenceIO did some

Kafka on yarn

2014-07-23 Thread hsy...@gmail.com
Hi guys, Kafka is getting more and more popular and in most cases people run kafka as long-term service in the cluster. Is there a discussion of running kafka on yarn cluster which we can utilize the convenient configuration/resource management and HA. I think there is a big potential and