So basically we're saying that even with a lot of tests, you're never sure to cover all the possible edge cases and the real stamp for "production readiness" is only when the "experimental features" have been deployed in various clusters with various scenarios/use-cases, just re-phrasing Blake here. Totally +1 on the idea.
Now I can foresee a problem with the "experimental" flag, that is nobody (in the community) will use it or even dare to play with it and thus the "experimental" features never get a chance to be tested and then we break the bug-reports/bug-fixes iterations ... How many times have I seen users on the ML asking which version of C* is the most fit for production and the answer was always at least 1 major version behind the current released major (2.1 was recommended when 3.x was released and so one ...) ? The fundamental issue here is that a lot of folks in the community do not want to take any risk and take a conservative approach for the production, which is fine and perfectly understandable. But it means that the implicit contract for OSS software, e.g. "you have a software for free in exchange you will give feedbacks and bug reports to improve it", is completely broken. Let's take the example of MV. MV was shipped with 3.0 --> considered not stable --> nobody/few people uses MV --> few bug reports --> bugs didn't have chance to get fixed --> the problem lasts until now About SASI, how many people really played with thoroughly apart from some toy examples ? Same causes, same consequences. And we can't even blame its design because fundamentally the architecture is pretty solid, just a question of usage and feedbacks. I suspect that this broken community QA/feedback loop did also explain partially the failure of tic/toc releases but it's only my own interpretation here. So if we don't figure out how to restore the "new feature/community bug report" strong feedback loop, we're going to face again the same issues and same debate in the future On Sun, Oct 1, 2017 at 5:30 PM, Blake Eggleston <beggles...@apple.com> wrote: > I'm not sure the main issue in the case of MVs is testing. In this case it > seems to be that there are some design issues and/or the design was only > works in some overly restrictive use cases. That MVs were committed knowing > these were issues seems to be the real problem. So in the case of MVs, sure > I don't think they should have ever made it to an experimental stage. > > Thinking of how an experimental flag fits in the with the project going > forward though, I disagree that we should avoid adding experimental > features. On the contrary, I think leaning towards classifying new features > as experimental would be better for users. Especially larger features and > changes. > > Even with well spec'd, well tested, and well designed features, there will > always be edge cases that you didn't think of, or you'll have made > assumptions about the other parts of C* it relies on that aren't 100% > correct. Small problems here can often affect correctness, or result in > data loss. So, I think it makes sense to avoid marking them as ready for > regular use until they've had time to bake in clusters where there are some > expert operators that are sophisticated enough to understand the > implications of running them, detect issues, and report bugs. > > Regarding historical examples, in hindsight I think committing 8099, or at > the very least, parts of it, behind an experimental flag would have been > the right thing to do. It was a huge change that we're still finding issues > with 2 years later. > > On October 1, 2017 at 6:08:50 AM, DuyHai Doan (doanduy...@gmail.com) > wrote: > > How should we transition one feature from the "experimental" state to > "production ready" state ? On which criteria ? > > > > On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson <krum...@gmail.com> > wrote: > > > I was just thinking that we should try really hard to avoid adding > > experimental features - they are experimental due to lack of testing > right? > > There should be a clear path to making the feature non-experimental (or > get > > it removed) and having that path discussed on dev@ might give more > > visibility to it. > > > > I'm also struggling a bit to find good historic examples of "this would > > have been better off as an experimental feature" - I used to think that > it > > would have been good to commit DTCS with some sort of experimental flag, > > but that would not have made DTCS any better - it would have been better > to > > do more testing, realise that it does not work and then not commit it at > > all of course. > > > > Does anyone have good examples of features where it would have made sense > > to commit them behind an experimental flag? SASI might be a good example, > > but for MVs - if we knew how painful they would be, they really would not > > have gotten committed at all, right? > > > > /Marcus > > > > On Sat, Sep 30, 2017 at 7:42 AM, Jeff Jirsa <jji...@gmail.com> wrote: > > > > > Reviewers should be able to suggest when experimental is warranted, and > > > conversation on dev+jira to justify when it’s transitioned from > > > experimental to stable? > > > > > > We should remove the flag as soon as we’re (collectively) confident in > a > > > feature’s behavior - at least correctness, if not performance. > > > > > > > > > > On Sep 29, 2017, at 10:31 PM, Marcus Eriksson <krum...@gmail.com> > > wrote: > > > > > > > > +1 on marking MVs experimental, but should there be some point in the > > > > future where we consider removing them from the code base unless they > > > have > > > > gotten significant improvement as well? > > > > > > > > We probably need to enforce some kind of process for adding new > > > > experimental features in the future - perhaps a mail like this one to > > > dev@ > > > > motivating why it should be experimental? > > > > > > > > /Marcus > > > > > > > > On Sat, Sep 30, 2017 at 1:15 AM, Vinay Chella > > > <vche...@netflix.com.invalid> > > > > wrote: > > > > > > > >> We tried perf testing MVs internally here but did not see good > results > > > with > > > >> it, hence paused its usage. +1 on tagging certain features which are > > not > > > >> PROD ready or not stable enough. > > > >> > > > >> Regards, > > > >> Vinay Chella > > > >> > > > >>> On Fri, Sep 29, 2017 at 7:22 PM, Ben Bromhead <b...@instaclustr.com > > > > > wrote: > > > >>> > > > >>> I'm a fan of introducing experimental flags in general as well, +1 > > > >>> > > > >>> > > > >>> > > > >>>> On Fri, 29 Sep 2017 at 13:22 Jon Haddad <j...@jonhaddad.com> > wrote: > > > >>>> > > > >>>> I’m very much +1 on this, and to new features in general. > > > >>>> > > > >>>> I think having a clear line in which we classify something as > > > >> production > > > >>>> ready would be nice. It would be great if committers were using > the > > > >>>> feature in prod and could vouch for it’s stability. > > > >>>> > > > >>>>> On Sep 29, 2017, at 1:09 PM, Blake Eggleston < > beggles...@apple.com > > > > > > >>>> wrote: > > > >>>>> > > > >>>>> Hi dev@, > > > >>>>> > > > >>>>> I’d like to propose that we retroactively classify materialized > > views > > > >>> as > > > >>>> an experimental feature, disable them by default, and require > users > > to > > > >>>> enable them through a config setting before using. > > > >>>>> > > > >>>>> Materialized views have several issues that make them > (effectively) > > > >>>> unusable in production. Some of the issues aren’t just > > implementation > > > >>>> problems, but problems with the design that aren’t easily fixed. > > It’s > > > >>>> unfair of us to make features available to users in this state > > without > > > >>>> providing a clear warning that bad or unexpected things are likely > > to > > > >>>> happen if they use it. > > > >>>>> > > > >>>>> Obviously, this isn’t great news for users that have already > > adopted > > > >>>> MVs, and I don’t have a great answer for that. I think that’s sort > > of > > > a > > > >>>> sunk cost at this point. If they have any MV related problems, > > they’ll > > > >>> have > > > >>>> them whether they’re marked experimental or not. I would expect > this > > > to > > > >>>> reduce the number of users adopting MVs in the future though, and > if > > > >> they > > > >>>> do, it would be opt-in. > > > >>>>> > > > >>>>> Once MVs reach a point where they’re usable in production, we can > > > >>> remove > > > >>>> the flag. Specifics of how the experimental flag would work can be > > > >>> hammered > > > >>>> out in a forthcoming JIRA, but I’d imagine it would just prevent > > users > > > >>> from > > > >>>> creating new MVs, and maybe log warnings on startup for existing > MVs > > > if > > > >>> the > > > >>>> flag isn’t enabled. > > > >>>>> > > > >>>>> Let me know what you think. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> > > > >>>>> Blake > > > >>>> > > > >>>> > > > >>>> ------------------------------------------------------------ > > --------- > > > >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org > > > >>>> > > > >>>> -- > > > >>> Ben Bromhead > > > >>> CTO | Instaclustr <https://www.instaclustr.com/> > > > >>> +1 650 284 9692 > > > >>> Reliability at Scale > > > >>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer > > > >>> > > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > >
