I am running on 0.8.1.1 and I thought that the partition reassignment tools can do this job. Just was not sure if this is the best way to do this. I will try this out in stage env first and will perform the same in prod.
Thanks, marcin On Mon, Aug 25, 2014 at 7:23 PM, Joe Stein <joe.st...@stealth.ly> wrote: > Marcin, that is a typical task now. What version of Kafka are you running? > > Take a look at > https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion > and > > https://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor > > Basically you can do a --generate to get existing JSON topology and with > that take the results of "Current partition replica assignment" (the first > JSON that outputs) and make whatever changes (like sed old node for new > node and add more replica's which increase the replication factor, whatever > you want) and then --execute. > > With lots of data this takes time so you will want to run --verify to see > what is in progress... good thing do a node at a time (even topic at a > time) however you want to manage and wait for it as such. > > The "preferred" replica is simply the first one in the list of replicas. > The kafka-preferred-replica-election.sh just makes that replica the leader > as this is not automatic yet. > > If you are running a version prior to 0.8.1.1 it might make sense to > upgrade the old nodes first then run reassign to the new servers. > > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > > On Mon, Aug 25, 2014 at 8:59 PM, Marcin Michalski <mmichal...@tagged.com> > wrote: > > > Hi, I would like to migrate my Kafka setup from old servers to new > servers. > > Let say I have 8 really old servers that have the kafka topics/partitions > > replicated 4 ways and want to migrate the data to 4 brand new servers and > > want the replication factor be 3. I wonder if anyone has ever performed > > this type of migration? > > > > Will auto rebalancing take care of this automatically if I do the > > following? > > > > Let say I bring down old broker id 1 down and startup new server broker > id > > 100 up, is there a way to migrate all of the data of the topic that had > the > > topic (where borker id 1 was the leader) over to the new broker 100? > > > > Or do I need to use *bin/kafka-preferred-replica-election.sh *to reassign > > the topics/partitions from old broker 1 to broker 100? And then just keep > > doing the same thing until all of the old brokers are decommissioned? > > > > Also, would kafka-preferred-replica-election.sh let me actually lower the > > number of replicas as well, if I just simply make sure that given > > topic/partition was only elected 3 times versus 4? > > > > Thanks for your insight, > > Marcin > > >
