This paper talks about how to do that - http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf It will be interesting to see what part of it Kafka can adopt, if any.
Thanks, Neha On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg <j...@squareup.com> wrote: > Letting each broker have a weight sounds like a great idea. > > Since in my use case, topics are generally auto-created, it won't be > practical to map brokers manually per topic. > > Thanks, > > Jason > > > On Fri, May 17, 2013 at 8:38 PM, Jun Rao <jun...@gmail.com> wrote: > > > In 0.8, you can create topics manually and explicitly specify the replica > > to broker mapping. Post 0.8, we can think of some more automated ways to > > deal with this (e.g., let each broker carry some kind of weight). > > > > Thanks, > > > > Jun > > > > > > On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > > > Hi, > > > > > > I'm wondering if there's a good way to have a heterogenous kafka > cluster > > > (specifically, if we have nodes with different sized disks). So, we > > might > > > want a larger node to receive more messages than a smaller node, etc. > > > > > > I expect there's something we can do with using a partitioner that has > > > specific knowledge about the hosts in the cluster, but this feels > messy, > > to > > > have this config on every producer client.... > > > > > > Thoughts? > > > > > > Jason > > > > > >
