Have you thought about integrating Kafka into a distributed resource management framework like Hadoop YARN (which would probably leverage HDFS) or Mesos? On May 23, 2013 11:31 PM, "Neha Narkhede" <neha.narkh...@gmail.com> wrote:
> This paper talks about how to do that - > http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf > It will be interesting to see what part of it Kafka can adopt, if any. > > Thanks, > Neha > > > On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > Letting each broker have a weight sounds like a great idea. > > > > Since in my use case, topics are generally auto-created, it won't be > > practical to map brokers manually per topic. > > > > Thanks, > > > > Jason > > > > > > On Fri, May 17, 2013 at 8:38 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > In 0.8, you can create topics manually and explicitly specify the > replica > > > to broker mapping. Post 0.8, we can think of some more automated ways > to > > > deal with this (e.g., let each broker carry some kind of weight). > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > Hi, > > > > > > > > I'm wondering if there's a good way to have a heterogenous kafka > > cluster > > > > (specifically, if we have nodes with different sized disks). So, we > > > might > > > > want a larger node to receive more messages than a smaller node, etc. > > > > > > > > I expect there's something we can do with using a partitioner that > has > > > > specific knowledge about the hosts in the cluster, but this feels > > messy, > > > to > > > > have this config on every producer client.... > > > > > > > > Thoughts? > > > > > > > > Jason > > > > > > > > > >
