On Thu, May 22, 2014 at 7:29 AM, Jun Rao <jun...@gmail.com> wrote:
> That's exactly what partition reassignment does. It may take a bit of time
> for the reassignment to complete.
>
> For example, if you initiate the process to change the replica assignment
> from {1, 2, 3} to {4,5,6}, the values in the assigned replica (AR) and
> leader/isr path in ZK may go through the following transition.
> AR leader/isr
> 1. {1,2,3} 1/{1,2,3}
> 2. {1,2,3,4,5,6} 1/{1,2,3}
> 3. {1,2,3,4,5,6} 1/{1,2,3,4,5,6}
> 4. {1,2,3,4,5,6} 4/{1,2,3,4,5,6}
> 5. {1,2,3,4,5,6} 4/{4,5,6}
> 6. {4,5,6} 4/{4,5,6}
>
> Going from step 2 to step 3 can take some time since we have to wait for
> the new replicas to fully catch up. The reassignment process completes when
> you are in step 6. The reassignment tool has a mode that you can use to
> check the status.
>
> Thanks,
>
Great! This is progress.
So I think some of mucking around I was doing caused the broker that I
wanted to drop the partitions to lose it's state.
Re-running execute w/ a json file that listed my target replicas (already
synced by now) caused that broker to drop the partitions and leave the
replica set finally.
> Jun
>
>
>
> On Wed, May 21, 2014 at 10:09 PM, David Birdsong
> <david.birds...@gmail.com>wrote:
>
> > On Wed, May 21, 2014 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote:
> >
> > > Expansion can be done by following
> > > http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
> > >
> > > If you just want to free up a server, you can stop the broker there and
> > > start a broker using the same broker id on a new server. Data should be
> > > automatically replicated to the new server.
> > >
> > >
> > The thing is, I don't want to fully free up the server, I just want it to
> > stop hosting all the topic,partition sets it originally did.
> >
> > Here's the flow:
> >
> > broker_1 hosts topic,{1,2,3,4}
> > over time load for the partitions overwhelms broker_1
> >
> > spin up broker_2
> > migrate topic,{3,4} to broker_2 successfully
> >
> > broker_1 is only useful if partitions topic,{3,4} are dropped by broker_1
> >
> > how does one get broker_1 to disassociate, drop, forget, evict...(not
> sure
> > the verb) topic,{3,4} and let broker_2 own it.
> >
> > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 21, 2014 at 3:23 PM, David Birdsong <
> > david.birds...@gmail.com
> > > >wrote:
> > >
> > > > Any suggestions? I'm kind of in a bind in that I don't understand how
> > to
> > > > grow the cluster when more capacity is needed--which happens to be
> > right
> > > > now for me.
> > > >
> > > > The only thing I can think that might work is to create new brokers,
> > > > cherry-pick topic/partition pairs and move them, then turn off the
> old
> > > ones
> > > > and forever retire their IDs freeing up my old hardware to come back
> > > online
> > > > as a new kafka broker ID.
> > > >
> > > >
> > > > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <
> > > david.birds...@gmail.com
> > > > >wrote:
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> > > > david.birds...@gmail.com>wrote:
> > > > >
> > > > >> Here's the reassignment json and current "describe" output:
> > > > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> > > > >>
> > > > >>
> > > > >> In my re-assignment json, I tried to re-assign to 2 when the repl
> is
> > > set
> > > > >> to 3. Once I noticed the the completely new node "133" had
> appeared
> > in
> > > > the
> > > > >> ISR, I tried stopping 224, wiping kafka completely and then
> brought
> > > 224
> > > > >> back up as 224 again. It promptly replicated the topic, but never
> > > joined
> > > > >> the ISR.
> > > > >>
> > > > >
> > > > > to be clear, i don't want it to join the ISR. i'm curious how to
> make
> > > 224
> > > > > forget about the partitions for that topic since i have other plans
> > for
> > > > it.
> > > > >
> > > > >
> > > > >>
> > > > >> How does one move a replica? This is exactly what I'm trying to
> do.
> > > > >>
> > > > >> My pattern is a common one. I started with a set of 3 kafka
> brokers.
> > > The
> > > > >> load and space is overwhelming them. I'm trying to add new brokers
> > and
> > > > >> spread the partitions to new nodes while removing some of the
> > > > partitions on
> > > > >> the old nodes so as to make room. It's the latter that I don't get
> > how
> > > > to
> > > > >> do.
> > > > >>
> > > > >> I've conflated two issues here mostly due to needing to get this
> > > cluster
> > > > >> stable again.
> > > > >> - reduce replication
> > > > >> - remove a partition from a broker, ie. remove the replica
> > > > >>
> > > > >> they're very distinct actions, but both would help me in the
> moment
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <jun...@gmail.com>
> wrote:
> > > > >>
> > > > >>> During the re-assignment, did you move the replica off the old
> > > broker?
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>>
> > > > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> > > > >>> david.birds...@gmail.com>wrote:
> > > > >>>
> > > > >>> > I did that and so now the topic has 4 replicas for a repl count
> > of
> > > 3,
> > > > >>> but
> > > > >>> > only the 'new' replicas exist in the ISR.
> > > > >>> >
> > > > >>> > The old broker that I want to clear disk space and generally
> free
> > > up
> > > > >>> > resources has fully synced a topic that I want to disassociate
> > from
> > > > it.
> > > > >>> >
> > > > >>> > Is there a way to do this?
> > > > >>> >
> > > > >>> >
> > > > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <jun...@gmail.com>
> > wrote:
> > > > >>> >
> > > > >>> > > We don't have an exact tool for doing this. You may be able
> to
> > do
> > > > >>> that
> > > > >>> > > through
> > > > >>> > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > > > >>> > > specifying fewer replicas.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > >
> > > > >>> > > Jun
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > > > >>> > david.birds...@gmail.com
> > > > >>> > > >wrote:
> > > > >>> > >
> > > > >>> > > > Is there a way to reduce the replication count? I'm trying
> to
> > > > >>> spread
> > > > >>> > > > existing partitions across more brokers, but it's hard to
> > > decomm
> > > > a
> > > > >>> > > broker.
> > > > >>> > > > Reducing repl count would suffice for now.
> > > > >>> > > >
> > > > >>> > > > Any tips?
> > > > >>> > > >
> > > > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
