One question I have is about whether/how to scope throttling to a reassignment. Currently throttles are only loosely associated with reassignment: You can start a reassignment without any throttling, add throttling to an in-flight reassignment, and remember/forget to remove throttling after the reassignment is complete. There's is great flexibility in that, but also the risk that you forget the remove the throttle(s).
Just adding an API for setting the throttled rate makes this situation worse: While it's nice to be able to auto-remove the throttles rate what about the config for the throttled replicas? Also you might add a throttle thinking a reassignment is in-flight, but it has in fact just finished: Those throttles will now hang around until reset or the end of the next reassignment. For these reasons it would be good if the throttle were more directly scoped to the reassignment. On the other hand, taking LinkedIn's Cruise Control as an example, there they seem to modify the reassignment znode directly and incrementally and so there is no notion of "the reassignment". Reassignments will be running continuously, with partitions added before all of the current partitions have completed. If there is no meaningful cluster-wide "reassignment" then it would be better to remove remove the throttle by changing the list of replicas as each replica catches up. I'm interested in any use cases people can share on this, as I'd like the throttle API to be useful for a broad range of use cases, rather than being too narrowly focussed on what's needed by the existing CLI tools. Thanks, Tom On 28 September 2017 at 17:22, Tom Bentley <t.j.bent...@gmail.com> wrote: > I'm starting to think about KIP-179 again. In order to have more > manageably-scoped KIPs and PRs I think it might be worth factoring-out the > throttling part into a separate KIP. Wdyt? > > Keeping the throttling discussion in this thread for the moment... > > The throttling behaviour is currently spread across the > `(leader|follower).replication.throttled.replicas` topic config and the > `(leader|follower).replication.throttled.rate` dynamic broker config. > It's not really clear to me exactly what "removing the throttle" is > supposed to mean. I mean we could reset the rate to Long.MAV_VALUE or we > could change the list of replicas to an empty list. The > ReassignPartitionsCommand does both, but there is some small utility in > leaving the rate, but clearing the list, if you've discovered the "right" > rate for your cluster/workload and to want it to be sticky for next time. > Does any one do this in practice? > > With regards to throttling, it would be >>> worth thinking about a way where the throttling configs can be >>> automatically removed without the user having to re-run the tool. >>> >> >> Isn't that just a matter of updating the topic configs for >> (leader|follower).replication.throttled.replicas at the same time we >> remove the reassignment znode? That leaves open the question about whether >> to reset the rates at the same time. >> > > Thinking some more about my "update the configs at the same time we remove > the reassignment znode" suggestion. The reassignment znode is persistent, > so the reassignment will survive a zookeeper restart. If there was a flag > for the auto-removal of the throttle it would likewise need to be > persistent. Otherwise a ZK restart would remember the reassignment, but > forget about the preference for auto removal of throttles. So, we would use > a persistent znode (a child of the reassignment path, perhaps) to store a > flag for throttle removal. > > Thoughts? > > Cheers, > > Tom >
