Any suggestions? I'm kind of in a bind in that I don't understand how to grow the cluster when more capacity is needed--which happens to be right now for me.
The only thing I can think that might work is to create new brokers, cherry-pick topic/partition pairs and move them, then turn off the old ones and forever retire their IDs freeing up my old hardware to come back online as a new kafka broker ID. On Wed, May 21, 2014 at 9:16 AM, David Birdsong <david.birds...@gmail.com>wrote: > > > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong > <david.birds...@gmail.com>wrote: > >> Here's the reassignment json and current "describe" output: >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5 >> >> >> In my re-assignment json, I tried to re-assign to 2 when the repl is set >> to 3. Once I noticed the the completely new node "133" had appeared in the >> ISR, I tried stopping 224, wiping kafka completely and then brought 224 >> back up as 224 again. It promptly replicated the topic, but never joined >> the ISR. >> > > to be clear, i don't want it to join the ISR. i'm curious how to make 224 > forget about the partitions for that topic since i have other plans for it. > > >> >> How does one move a replica? This is exactly what I'm trying to do. >> >> My pattern is a common one. I started with a set of 3 kafka brokers. The >> load and space is overwhelming them. I'm trying to add new brokers and >> spread the partitions to new nodes while removing some of the partitions on >> the old nodes so as to make room. It's the latter that I don't get how to >> do. >> >> I've conflated two issues here mostly due to needing to get this cluster >> stable again. >> - reduce replication >> - remove a partition from a broker, ie. remove the replica >> >> they're very distinct actions, but both would help me in the moment >> >> >> >> >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <jun...@gmail.com> wrote: >> >>> During the re-assignment, did you move the replica off the old broker? >>> >>> Thanks, >>> >>> Jun >>> >>> >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong < >>> david.birds...@gmail.com>wrote: >>> >>> > I did that and so now the topic has 4 replicas for a repl count of 3, >>> but >>> > only the 'new' replicas exist in the ISR. >>> > >>> > The old broker that I want to clear disk space and generally free up >>> > resources has fully synced a topic that I want to disassociate from it. >>> > >>> > Is there a way to do this? >>> > >>> > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <jun...@gmail.com> wrote: >>> > >>> > > We don't have an exact tool for doing this. You may be able to do >>> that >>> > > through >>> > > >>> > > >>> > >>> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby >>> > > specifying fewer replicas. >>> > > >>> > > Thanks, >>> > > >>> > > Jun >>> > > >>> > > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong < >>> > david.birds...@gmail.com >>> > > >wrote: >>> > > >>> > > > Is there a way to reduce the replication count? I'm trying to >>> spread >>> > > > existing partitions across more brokers, but it's hard to decomm a >>> > > broker. >>> > > > Reducing repl count would suffice for now. >>> > > > >>> > > > Any tips? >>> > > > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.) >>> > > > >>> > > >>> > >>> >> >> >
