I think using feature flags (whether we support a framework and tooling for feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to this KIP. I think the value of this KIP is that it's trying to propose a systemic approach for gating functionality that may take multiple releases to develop. A problem with ad-hoc feature flags is that it's useful during feature development, so that people who are working on this feature (or are interested in beta-testing the feature) can get early access (without any guarantees on compatibility or even correctness); but then the feature flags often stick forever after the feature development is done (and as time moves one, the new code is written in such a way that it's not possible to turn the feature off any more cleanly).
I'd also clarify how I think about "stable". Ismael made a comment " something is stable in the "this is battle-tested" sense.". I don't think it has to be "battle-tested", it just has to meet the bar of being in the trunk. Again, thinking of a small single-commit feature -- to commit to trunk, the feature doesn't have to be "battle-tested", but it should be complete (and not just a bunch of TODOs), with tests written and some level of dev-testing done, so that once the release is cut, we can find and fix bugs and make it release-quality (as opposed to reverting the whole thing). We can apply the same "stability" bar for features to be in the stable MV -- fully complete, tests written and some level of dev testing done. -Artem On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan <jols...@confluent.io.invalid> wrote: > Hi Ismael, > > I debated including something about feature flags in my last comment, but > maybe I should have. > What you said makes sense. > > Justine > > On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma <m...@ismaeljuma.com> wrote: > > > Justine, > > > > For features that are not production-ready, they should have an > additional > > configuration (not the metadata version) that enables/disables it. The MV > > specific features we ship are something we have to support and make sure > we > > don't break going forward. > > > > Ismael > > > > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan > > <jols...@confluent.io.invalid> > > wrote: > > > > > Hi Ismael, > > > > > > I think the concern I have about a MV for a feature that is not > > production > > > ready is that it blocks any development/features with higher MV > versions > > > that could be production ready. > > > > > > I do see your point though. Previously MV/IBP was about pure broker > > > compatibility and not about the confidence in the feature it is > gating. I > > > do wonder though if it could be useful to have that sort of gating. > > > I also wonder if an internal config could be useful so that the average > > > user doesn't have to worry about the complexities of unstable metadata > > > versions (and their risks). > > > > > > I am ok with options 2 and 2 as well by the way. > > > > > > Justine > > > > > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma <m...@ismaeljuma.com> wrote: > > > > > > > Hi, > > > > > > > > Thanks for the KIP. > > > > > > > > Reading the discussion, I think a lot of the confusion is due to the > > > > "unstable" naming. People are then trying to figure out when we think > > > > something is stable in the "this is battle-tested" sense. But this > flag > > > > should not be about that. We can have an MV for a feature that is not > > yet > > > > production-ready (and we did that when KRaft itself was not > production > > > > ready). I think this flag is about metadata versions that are > basically > > > > unsupported - if you use it, you get to keep the pieces. They exist > > > solely > > > > to make the lives of Apache Kafka developers easier. I would even > > suggest > > > > that the config we introduce be of the internal variety, ie it won't > > show > > > > in the generated documentation and there won't be any compatibility > > > > guarantee. > > > > > > > > Thoughts? > > > > > > > > Ismael > > > > > > > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano > > > > <pprovenz...@confluent.io.invalid> wrote: > > > > > > > > > Hey folks, > > > > > > > > > > I am starting a discussion thread for managing unstable metadata > > > versions > > > > > in Apache Kafka. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > > > > > > > This KIP is actually already implemented in 3.7 with PR > > > > > https://github.com/apache/kafka/pull/14860. > > > > > I have created this KIP to explain the motivation and how managing > > > > Metadata > > > > > Versions is expected to work. > > > > > Comments are greatly appreciated as this process can always be > > > improved. > > > > > > > > > > -- > > > > > --Proven > > > > > > > > > > > > > > >
