As Ewen suggests, you might want to take a look at KIP-6. The only thing to be careful about is not moving data for partitions other than those belonging to the topics whose replication factor is being changed by the tool.
It seems logical to expose this via the kafka-topics --alter option. Thanks, Neha On Mon, Mar 9, 2015 at 9:55 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Geoff, > > First, if you haven't already, take a look at > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing > and the associated JIRA. > > My opinion is that we shouldn't do anything to the existing partitions -- > it would probably violate the principle of least surprise if you started > moving data around when all I asked you to do was add new partitions. > > I think this aligns well with the basic approach of KIP 6 (which is not > approved yet, but I think folks are generally in favor of once it is fully > specified). The goal is to move as little data as possible, but achieve the > best balance. I think if the user wanted any *other* partitions rebalanced, > they should use the reassignment tool, which KIP 6 will hopefully make a > lot easier soon. But at a minimum, adding the new partitions should try not > to make the balance any worse than it already is. > > Not sure what this means for fixing the JIRA though -- reusing what KIP 6 > is doing but applying it to the new partitions seems like the ideal > solution. The current createTopic code doesn't take into account existing > assignments, but I don't think it makes sense to duplicate effort between > the two patches. I'm not sure what the timeframe of KIP 6, but if its not > too far away you might just mark it as blocking KAFKA-1313, prepare a patch > against the KAFKA-1792's latest version, and maybe update KAFKA-1313's fix > version to 0.8.3 to match KAFKA-1792's. > > -Ewen > > On Mon, Mar 9, 2015 at 3:14 PM, Geoffrey Anderson <ge...@confluent.io> > wrote: > > > Hi dev list, I have a few questions regarding KAFKA-1313 ( > > https://issues.apache.org/jira/browse/KAFKA-1313) > > > > It is currently possible albeit in a kindof hacky way to increase > > replication factor on a topic by using the partition reassignment tool > with > > a hand-crafted reassignment plan. > > > > The preferred approach to making this better is to add support for > > replication-factor in the alter-topic command. For example, > > > > bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --alter > > --replication-factor 2 > > > > > > Another approach is use the partition reassignment tool to auto-generate > a > > reassignment plan which results in increased replication. > > > > The advantage of the alter topics approach is that it seems logical to > > group this functionality with the other alter topic stuff. However, a > > couple questions: > > - Should partitions automatically be reassigned to brokers and shuffled > > when the replication factor is changed? > > - If so, it would be best to be able to update the replication factor of > > multiple topics at once so that the automatic reassignment would take > place > > only once for the whole batch of changes. > > > > One advantage of the partition reassignment approach is that it's easier > to > > generate a reassignment plan which updates multiple topics at once, and > you > > have a chance to review the reassignment plan before putting it into > effect > > with the --execute flag. > > > > Thanks, > > Geoff > > > > > > -- > Thanks, > Ewen > -- Thanks, Neha
