Expansion can be done by following http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
If you just want to free up a server, you can stop the broker there and start a broker using the same broker id on a new server. Data should be automatically replicated to the new server. Thanks, Jun On Wed, May 21, 2014 at 3:23 PM, David Birdsong <david.birds...@gmail.com>wrote: > Any suggestions? I'm kind of in a bind in that I don't understand how to > grow the cluster when more capacity is needed--which happens to be right > now for me. > > The only thing I can think that might work is to create new brokers, > cherry-pick topic/partition pairs and move them, then turn off the old ones > and forever retire their IDs freeing up my old hardware to come back online > as a new kafka broker ID. > > > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <david.birds...@gmail.com > >wrote: > > > > > > > > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong < > david.birds...@gmail.com>wrote: > > > >> Here's the reassignment json and current "describe" output: > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5 > >> > >> > >> In my re-assignment json, I tried to re-assign to 2 when the repl is set > >> to 3. Once I noticed the the completely new node "133" had appeared in > the > >> ISR, I tried stopping 224, wiping kafka completely and then brought 224 > >> back up as 224 again. It promptly replicated the topic, but never joined > >> the ISR. > >> > > > > to be clear, i don't want it to join the ISR. i'm curious how to make 224 > > forget about the partitions for that topic since i have other plans for > it. > > > > > >> > >> How does one move a replica? This is exactly what I'm trying to do. > >> > >> My pattern is a common one. I started with a set of 3 kafka brokers. The > >> load and space is overwhelming them. I'm trying to add new brokers and > >> spread the partitions to new nodes while removing some of the > partitions on > >> the old nodes so as to make room. It's the latter that I don't get how > to > >> do. > >> > >> I've conflated two issues here mostly due to needing to get this cluster > >> stable again. > >> - reduce replication > >> - remove a partition from a broker, ie. remove the replica > >> > >> they're very distinct actions, but both would help me in the moment > >> > >> > >> > >> > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <jun...@gmail.com> wrote: > >> > >>> During the re-assignment, did you move the replica off the old broker? > >>> > >>> Thanks, > >>> > >>> Jun > >>> > >>> > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong < > >>> david.birds...@gmail.com>wrote: > >>> > >>> > I did that and so now the topic has 4 replicas for a repl count of 3, > >>> but > >>> > only the 'new' replicas exist in the ISR. > >>> > > >>> > The old broker that I want to clear disk space and generally free up > >>> > resources has fully synced a topic that I want to disassociate from > it. > >>> > > >>> > Is there a way to do this? > >>> > > >>> > > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <jun...@gmail.com> wrote: > >>> > > >>> > > We don't have an exact tool for doing this. You may be able to do > >>> that > >>> > > through > >>> > > > >>> > > > >>> > > >>> > http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby > >>> > > specifying fewer replicas. > >>> > > > >>> > > Thanks, > >>> > > > >>> > > Jun > >>> > > > >>> > > > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong < > >>> > david.birds...@gmail.com > >>> > > >wrote: > >>> > > > >>> > > > Is there a way to reduce the replication count? I'm trying to > >>> spread > >>> > > > existing partitions across more brokers, but it's hard to decomm > a > >>> > > broker. > >>> > > > Reducing repl count would suffice for now. > >>> > > > > >>> > > > Any tips? > >>> > > > > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.) > >>> > > > > >>> > > > >>> > > >>> > >> > >> > > >
