Well if you're willing to manage the partitions by yourself, I think it would be possible. You would have to create more partitions for your topics and then assign more partitions to the new machines.
If kafka was able to rebalance automaticaly, and doing this by not only considering the number of machine but the performance of those, it would be a really nice to have. Francis Sent from Samsung Mobile -------- Original message -------- From: Jason Rosenberg <j...@squareup.com> Date: 05-17-2013 6:26 PM (GMT-05:00) To: users@kafka.apache.org Subject: Re: heterogenous kafka cluster? Yeah, I thought of that (running 2 kafkas on one box), but it doesn't really add the benefit of redundancy through replication (e.g. if we have 2 replicas mapping to the same physical machine). Jason On Fri, May 17, 2013 at 2:50 PM, Chris Riccomini <criccom...@linkedin.com>wrote: > Hey guys, > > I have no idea if this would be reasonable, but what about just running > two Kafka processes on the bigger box? > > Cheers, > Chris > > On 5/17/13 2:48 PM, "Jason Rosenberg" <j...@squareup.com> wrote: > > >Just resource allocation issues. E.g. imagine having an existing kafka > >cluster with one machine spec, and getting access to a few more hosts to > >augment the cluster, which are newer and therefore have twice the disk > >storage. I'd like to seamlessly add them into the cluster, without having > >to replace everything en masse. Thus, it would be nice for the newer ones > >to take proportionally more load based on the relative storage available, > >etc. > > > >Jason > > > > > >On Fri, May 17, 2013 at 2:34 PM, Neha Narkhede > ><neha.narkh...@gmail.com>wrote: > > > >> That does seem a little hacky. But I'm trying to understand the > >>requirement > >> behind having to deploy heterogeneous hardware. What are you trying to > >> achieve or optimize? > >> > >> Thanks, > >> Neha > >> > >> > >> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <j...@squareup.com> > >>wrote: > >> > >> > Hi, > >> > > >> > I'm wondering if there's a good way to have a heterogenous kafka > >>cluster > >> > (specifically, if we have nodes with different sized disks). So, we > >> might > >> > want a larger node to receive more messages than a smaller node, etc. > >> > > >> > I expect there's something we can do with using a partitioner that has > >> > specific knowledge about the hosts in the cluster, but this feels > >>messy, > >> to > >> > have this config on every producer client.... > >> > > >> > Thoughts? > >> > > >> > Jason > >> > > >> > >
