There are folks that run Kafka Brokers on Apache Mesos. I don't know of anyone running Kafka brokers on YARN but if there were I would hope they chime in.
Without getting into a long debate about Mesos vs YARN I do agree with cluster resource allocation being an important direction for the industry as a whole. There was a discussion some time back about this on the dev list (maybe last November) and there isn't really anything that Kafka needs to-do for making this work. I had started that discussion and a wiki page and at the end of it was able to get Kafka running without any changes to it. Each of these resource negotiator systems provide ways for standalone systems to run on them to avoid custom schedulers from having to be developed. Their (Mesos/YARN/etc) success is somewhat dictated by not forcing application to have to-do much of anything to gain from the benefits they provide =8^) If you are interested in running Kafka Brokers on Mesos you can do so using the Apache Aurora scheduler. Some scripts that will launch that for you on a Mesos cluster can be found here https://github.com/pegasussolutions/borealis (along with the zookeeper ensemble if you wanted). I know folks also use Marathon (another Mesos schedular) for this too but I don't know if those scripts were open sourced or not but it is possible. Running consumers on Mesos and/or YARN has become more prevalent (almost typical) with systems like Spark, Samza and Storm taking up the data processing from brokers and those systems being launched on Mesos and/or YARN clusters. Producers also but that falls a bit more into just running whatever application that is producing data on Mesos and/or YARN and your being able to produce to Kafka brokers from within that application. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Wed, Jul 23, 2014 at 5:37 PM, hsy...@gmail.com <hsy...@gmail.com> wrote: > Hi guys, > > Kafka is getting more and more popular and in most cases people run kafka > as long-term service in the cluster. Is there a discussion of running kafka > on yarn cluster which we can utilize the convenient configuration/resource > management and HA. I think there is a big potential and requirement for > that. > I found a project https://github.com/kkasravi/kafka-yarn. But is there a > official roadmap/plan for this? > > Thank you very much! > > Best, > Siyuan >
