I would not think that partitions moving would cause any orphaned messages like that. I would be more concerned about what happens when you change the default on a running cluster from one scheme to another. Would we want to support some kind of automated reassignment of existing partitions (personally - no. I want to trigger that manually because it is a very disk and network intensive process)?
-Todd On Wed, Mar 4, 2015 at 7:33 PM, Tong Li <liton...@us.ibm.com> wrote: > > > Todd, > I think plugable design is good with solid default. The only issue I > feel is when you use one and switch to another, will we end up with some > unread messages hanging around and no one thinks or knows it is their > responsibility to take care of them? > > Thanks. > > Tong > > Sent from my iPhone > > > On Mar 5, 2015, at 10:46 AM, Todd Palino <tpal...@gmail.com> wrote: > > > > Apologize for the late comment on this... > > > > So fair assignment by count (taking into account the current partition > > count of each broker) is very good. However, it's worth noting that all > > partitions are not created equal. We have actually been performing more > > rebalance work based on the partition size on disk, as given equal > > retention of all topics, the size on disk is a better indicator of the > > amount of traffic a partition gets, both in terms of storage and network > > traffic. Overall, this seems to be a better balance. > > > > In addition to this, I think there is very much a need to have Kafka be > > rack-aware. That is, to be able to assure that for a given cluster, you > > never assign all replicas for a given partition in the same rack. This > > would allow us to guard against maintenances or power failures that > affect > > a full rack of systems (or a given switch). > > > > I think it would make sense to implement the reassignment logic as a > > pluggable component. That way it would be easy to select a scheme when > > performing a reassignment (count, size, rack aware). Configuring a > default > > scheme for a cluster would allow for the brokers to create new topics and > > partitions in compliance with the requested policy. > > > > -Todd > > > > > > On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > I will go back through the ticket and code and write more up. Should be > > > able to-do that sometime next week. The intention was to not replace > > > existing functionality by issue a WARN on use. The following version it > is > > > released we could then deprecate it... I will fix the KIP for that too. > > > > > > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <n...@confluent.io> > wrote: > > > > > > > Hey Joe, > > > > > > > > 1. Could you add details to the Public Interface section of the KIP? > This > > > > should include the proposed changes to the partition reassignment > tool. > > > > Also, maybe the new option can be named --rebalance instead of > > > > --re-balance? > > > > 2. It makes sense to list --decommission-broker as part of this KIP. > > > > Similarly, shouldn't we also have an --add-broker option? The way I > see > > > > this is that there are several events when a partition reassignment > is > > > > required. Before this functionality is automated on the broker, the > tool > > > > will generate an ideal replica placement for each such event. The > users > > > > should merely have to specify the nature of the event e.g. adding a > > > broker > > > > or decommissioning an existing broker or merely rebalancing. > > > > 3. If I understand the KIP correctly, the upgrade plan for this > feature > > > > includes removing the existing --generate option on the partition > > > > reassignment tool in 0.8.3 while adding all the new options in the > same > > > > release. Is that correct? > > > > > > > > Thanks, > > > > Neha > > > > > > > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <jay.kr...@gmail.com> > wrote: > > > > > > > > > Ditto on this one. Can you give the algorithm we want to implement? > > > > > > > > > > Also I think in terms of scope this is just proposing to change the > > > logic > > > > > in ReassignPartitionsCommand? I think we've had the discussion > various > > > > > times on the mailing list that what people really want is just for > > > Kafka > > > > to > > > > > do it's best to balance data in an online fashion (for some > definition > > > of > > > > > balance). i.e. if you add a new node partitions would slowly > migrate to > > > > it, > > > > > and if a node dies, partitions slowly migrate off it. This could > > > > > potentially be more work, but I'm not sure how much more. Has > anyone > > > > > thought about how to do it? > > > > > > > > > > -Jay > > > > > > > > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <joe.st...@stealth.ly> > > > > wrote: > > > > > > > > > > > Posted a KIP for --re-balance for partition assignment in > > > reassignment > > > > > > tool. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New > +reassignment+partition+logic+for+re-balancing > > > > > > > > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792 > > > > > > > > > > > > While going through the KIP I thought of one thing from the JIRA > that > > > > we > > > > > > should change. We should preserve --generate to be existing > > > > functionality > > > > > > for the next release it is in. If folks want to use --re-balance > then > > > > > > great, it just won't break any upgrade paths, yet. > > > > > > > > > > > > /******************************************* > > > > > > Joe Stein > > > > > > Founder, Principal Consultant > > > > > > Big Data Open Source Security LLC > > > > > > http://www.stealth.ly > > > > > > Twitter: @allthingshadoop > <http://www.twitter.com/allthingshadoop> > > > > > > ********************************************/ > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Neha > > > > > > > >
