On Thu, Jan 23, 2025 at 3:20 PM Dmitry Konstantinov <netud...@gmail.com> wrote:
> Hi Stefan, > > Thank you a lot for the detailed feedback! Few comments: > > >> I think this is already the case, more or less. We are not doing perf > changes in older branches. > Yes, I understand the idea about stability of older branches, the primary > issue for me is that if I contribute even a small improvement to trunk - I > cannot really use it for a long time (except having it in my own > fork), because there is no release to get it back for me or anybody else.. > > >> Maybe it would be better to make the upgrading process as smooth as > possible so respective businesses are open to upgrade their clusters in a > more frequent manner. > About the upgrade process: my personal experience (3.0.x -> 3.11.x -> > 4.0.x -> 4.1.x), the upgrade in Cassandra is positive (I suppose the > autotests which test it are really helpful), I have not experienced any > serious issues with it. I suppose the majority of time when people have an > issue with upgrades is due to delaying them for too long and staying on > very old unsupported versions till the last moment. > > >> Cassandra is not JDK. We need to fix bugs in older branches we said we > support > Regarding the necessity to support the older branches it is the same story > for JDK: they now support and fix bugs in JDK8, JDK11, JDK17 and JDK 21 as > LTS versions and JDK23 as the latest release while developing and releasing > JDK24 now. > That is ... 6 branches at once. We were there, 3.0, 3.11, 4.0, 4.1, 5.0, trunk. If there was a bug in 3.0, because we were supporting that, we had to put this into 6 branches. That means 6 builds in CI. Each CI takes a couple hours ... If there is something wrong or the patch is changed we need to rebuild. So what looks like "just merge up from 3.0 and that's it" becomes a multi-day odyssey somebody needs to invest resources into. As we dropped 3.0 and 3.11 and we took care of 4.0+ that is better but still not fun when done "at scale". > Another example, Postgres does a major release every year: > https://www.postgresql.org/support/versioning/ and supports the last 5 > major versions. > Yeah, but they have most probably way more man-power as well etc ... > > >> please keep in mind that there are people behind the releases who are > spending time on that. > Yes, as I already mentioned, I really thank you to Brandon and Mick for > doing it! It is hard, exhausting and not the most exciting work to do. > Please contact me if I can help somehow with it, like checking and fixing > CI test failures(I've already done it for a while) / doing some scripting/ > etc. > I have a hypothesis (maybe I am completely wrong here) that actually the > low interest in the releasing process is somehow related to having a > Cassandra fork by many contributors, so there is no big demand for regular > mainline releases if you have them in a fork.. > > Regards, > Dmitry > > > > > > > > > > On Thu, 23 Jan 2025 at 12:30, Štefan Miklošovič <smikloso...@apache.org> > wrote: > >> I think the current guidelines are sensible. >> >> Going through your suggestions: >> >> 1) I think this is already the case, more or less. We are not doing perf >> changes in older branches. This is what we see in CASSANDRA-19429, a user >> reported that it is a performance improvement, and most probably he is >> right, but I am hesitant to refactor / introduce changes into older >> branches. >> >> Cassandra has a lot of inertia, we can not mess with what works even >> performance improvements are appealing. Maybe it would be better to make >> the upgrading process as smooth as possible so respective businesses are >> open to upgrade their clusters in a more frequent manner. >> >> 2) Well, but Cassandra is not JDK. We need to fix bugs in older branches >> we said we support. This is again related to inertia Cassandra has as a >> database. Bug fixes are always welcome, especially if there is 0 risk >> deploying it. >> >> What particularly resonates with me is your wording "more frequent and >> predictable". Well ... I understand it would be the most ideal outcome, but >> please keep in mind that there are people behind the releases who are >> spending time on that. I have been following this project for a couple >> years and the only people who are taking care of releases are Brandon and >> Mick. I was helping here and there to at least stage it and I am willing to >> continue to do so, but that is basically it. "two and a half" people are >> doing releases. For all these years. >> >> So if you ask for more frequent releases, that is something which is >> going to directly affect respective people involved in them. I guess they >> are doing it basically out of courtesy and it would be great to see more >> PMCs involved in release processes. As of now, it looks like everybody just >> assumes that "it will be somehow released" and "releases just happen" but >> that is not the case. Releases are not "just happening". There are people >> behind them who need to plan when it is going to happen and they need to >> find time for that etc. There are a lot of things not visible behind the >> scenes and doing releases is a job in itself. >> >> So if we ask for more frequent releases, it is a good question to ask who >> would be actually releasing that. >> >> On Wed, Jan 22, 2025 at 12:17 PM Dmitry Konstantinov <netud...@gmail.com> >> wrote: >> >>> Hi all, >>> >>> I am one of the contributors for the recent perf changes, like: >>> https://issues.apache.org/jira/browse/CASSANDRA-20165 >>> https://issues.apache.org/jira/browse/CASSANDRA-20226 >>> https://issues.apache.org/jira/browse/CASSANDRA-19557 >>> ... >>> >>> My motivation: I am currently using 4.1.x and planning to adopt 5.0.x in >>> the next quarter. Of course, I want to have it in the best possible share >>> from performance point of view, performance is one of important selling >>> points for upgrades. In general, performance is one of key reasons why >>> people select NoSQL and Cassandra particularly, so any improvement here >>> should be appreciated by users, especially in the current cloud-oriented >>> world where every such improvement is a potential cost saving. >>> >>> For me the question is tightly related to the release scheduling. We >>> have periodic and quite frequent patch releases now, thank you a lot to the >>> people who spend their time to do it. When we speak about minor releases - >>> it looks like the release process is much slower and not so predictable, it >>> can be a year or even more before I can get any minor release which >>> includes a change, and nobody can say even a preliminary date for it. >>> As a result when I have a performance patch and it is suggested to merge >>> only to trunk I will not get the improvement back to use for a long time. >>> So, I have 2 options in this case: >>> 1) relax and wait (potentially losing an interest due to a delayed >>> feedback) >>> 2) keep my own private fork to accumulate such changes with >>> correspondent overheads (what I am actually do now) >>> >>> As a guy who supports Cassandra in production for systems with 99.999 >>> availability requirements, of course I am curious about stability too, but >>> I think we need some balance here and we should rely more on things like >>> test coverage and different policies for different branches to not stagnate >>> due to fear of any change. I am not saying about massive breaking changes, >>> especially which modify (even in a compatible way) network communication >>> protocols or disk data formats, it should be a separate individual >>> discussion for them. >>> >>> The situation reminds me of the story of JDK prior to Java 9. There were >>> also some big bang minor releases (1.5/1.6/1.7/1.8) which we waited for a >>> very long time and Java was evolving very slowly. Now we have a model where >>> a new release is available every 1/2 year and some of them are supported as >>> long term. So, the people who prefer stability select and use LTS versions, >>> the people who want to get access to new features/improvements can take the >>> latest release, all are happy. Similar models like stable/latest releases >>> are available for other products. >>> >>> So, my suggestion is one of the following options: >>> 1) Classify the current release branches as more and less stable, like: >>> -- 4.0.x/4.1.x - avoid perf changes unless it is really a bug-like >>> -- 5.0.x - more relaxed rules >>> >>> 2) Do something similar to JDK with LTS versions: make minor releases >>> for the latest major version (like: 5.1/5.2) more frequent and predictable, >>> like a train release, do not create a fix branch for every one, >>> periodically for some selected minor versions establish fix branches and >>> release patch versions for them. >>> >>> Thank you, >>> Dmitry >>> >>> >>> On Wed, 22 Jan 2025 at 09:02, Jeff Jirsa <jji...@gmail.com> wrote: >>> >>>> I think the status quo is fine - perf goes to trunk, if you think >>>> something is special, it goes to the mailing list to justify exceptions >>>> >>>> >>>> On Jan 22, 2025, at 3:36 AM, Jordan West <jw...@apache.org> wrote: >>>> >>>> >>>> Thanks for the initial feedback. I hear a couple different themes / >>>> POVs. >>>> >>>> David/Paulo, it sounds like maybe a guide for perf backports + mailing >>>> list consensus when necessary + clear documentation of this could be a way >>>> forward. I agree that each change comes with stability risks but at the >>>> same time the greatest stability risk with Cassandra historically has been >>>> major version upgrades (although we have made great improvements here). For >>>> folks who want only the performance improvements, we are asking them to >>>> take greater risk by upgrading a major version or to maintain a fork. The >>>> fork is reasonable for some of the larger operators but not others. That >>>> said, I do agree we need to use judgement. Not all changes are worth >>>> backporting and some may incur too much risk. We could also add to the >>>> guide suggestions of how to de-risk a change (e.g. code is isolated, config >>>> to turn it off / off by default, etc). >>>> >>>> Jeff, I agree 1% wins aren't worth it if they are invasive and in risky >>>> areas. Not all of the improvements are that minor. >>>> >>>> Jordan >>>> >>>> On Tue, Jan 21, 2025 at 1:57 PM Jeff Jirsa <jji...@gmail.com> wrote: >>>> >>>>> We expect users to treat patch and minor releases as low risk. >>>>> Changing something deep in the storage engine to be 1% faster is not worth >>>>> the risk, because most users will skip the type of qualification that >>>>> finds >>>>> those one in a billion regressions. >>>>> >>>>> Patch releases are for bug fixes not perf improvements. >>>>> >>>>> >>>>> On Jan 21, 2025, at 9:10 PM, Jordan West <jw...@apache.org> wrote: >>>>> >>>>> >>>>> Hi folks, >>>>> >>>>> A topic that’s come up recently is what branches are valid targets for >>>>> performance improvements. Should they only go into trunk? This has come up >>>>> in the context of BTI improvements, Dmitry’s work on reducing object >>>>> overhead, and my work on CASSANDRA-15452. >>>>> >>>>> We currently have guidelines published: >>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199530302#Patching,versioning,andLTSreleases-Wheretoapplypatches. >>>>> But there’s no explicit discussion of how to handle performance >>>>> improvements. We tend to discuss whether they’re “bugfixes”. >>>>> >>>>> I’d like to discuss whether performance improvements should target >>>>> more than just trunk. I believe they should target every active branch >>>>> because performance is a major selling point of Cassandra. It’s not >>>>> practical to ask users to upgrade major versions for simple performance >>>>> wins. A major version can be deployed for years, especially when the next >>>>> one has major changes. But we shouldn’t target non-supported major >>>>> versions, either. Also, there will be exceptions: patches that are too >>>>> large, invasive, risky, or complicated to backport. For these, we rely on >>>>> the contributor and reviewer’s judgment and the mailing list. So, I’m >>>>> proposing an allowance to backport to active branches, not a requirement >>>>> to >>>>> merge them. >>>>> >>>>> I’m curious to hear your thoughts. >>>>> Jordan >>>>> >>>>> >>> >>> -- >>> Dmitry Konstantinov >>> >> > > -- > Dmitry Konstantinov >
