Thanks Stanislav and Colin for the feedback. I've updated the KIP to make it simpler. It's not updating the CreateTopics/CreatePartitions RPCs anymore. I've kept the broker setting so admins can keep the current behaviour but simplified it to be either enabled or disabled.
I've also kept the observed_brokers nodes in Zookeeper. I can't think of a better alternative to keep track of the expected brokers. The other option would be to perform the extra replica creation asynchronously (driven by the controller when a broker joins the cluster) but that feels a lot more complicated for this specific use case. I've also made it explicit that at least "min.insync.replicas" brokers have to be online to allow topic/partition creation. Thanks On Mon, Mar 25, 2019 at 1:17 PM Mickael Maison <mickael.mai...@gmail.com> wrote: > > Thanks Colin for the feedback. > > The idea was to allow both users and administrator to decide if they > wanted to opt-in and if so under what conditions. > > Maybe we could do something simpler and just allow the creation if at > least min-in-sync replicas are available? That should not require > changes to the protocol and while this might not cover all possible > use cases, that would still cover the use cases we've listed in the > KIP. That would also tie in with existing semantics/guarantees > (min-in-sync). > > Thanks > > On Tue, Feb 26, 2019 at 5:40 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > Hi Mickael, > > > > I don't think adding CREATED_UNDER_REPLICATED as an error code makes sense. > > It is not an error condition, as described here. > > > > > Updates to the Decommissioning brokers section in the documentation > > > will mention that if a broker id is never to be reused then its > > > corresponding node in zookeeper > > > /brokers/observed_ids will need to be removed manually > > > > I don't think it's acceptable to ask admins to manually modify ZooKeeper > > here. In general the ZK changes seem kind of like a hack -- perhaps we > > should drop it from the proposal for now. > > > > Perhaps we could even somehow do all of this in a custom CreateTopicPolicy? > > That would avoid the need for RPC changes, new configuration knobs, etc. > > > > best, > > Colin > > > > > > On Tue, Dec 18, 2018, at 08:43, Mickael Maison wrote: > > > Hi, > > > > > > We have submitted a KIP to handle topics and partitions creation when > > > a cluster is not fully available: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions > > > > > > As always, we welcome feedback and suggestions. > > > > > > Thanks > > > Mickael and Edoardo > > >
