I agree there’s probably some value captured in having ccm tests, but I suspect not in most of the python tests we have, which are all but unmaintained at this point. When I looked at them last I was surprised at how rubbish many of them were.
I think we need a plan to maintain them properly, and ensure they actually offer some value. Personally my proposal on that front is that we get rid of them (except if demonstrably testing some unique edge case) and instead run Harry against a ccm cluster we upgrade during the run. > On 28 Jan 2025, at 16:16, Mick Semb Wever <m...@apache.org> wrote: > > > replies inline… > . > > >>> We have far fewer (and more effective?) JVM Upgrade DTests. >>> There we only need 8x medium (3 cpu, 5GB ram) servers >> Does anyone have a strong understanding of the coverage and value offered by >> the python upgrade dtests vs. the in-jvm dtests? I don't, but I intuitively >> have a hard time believing the value difference matches the hardware >> requirement difference there. > > > I believe they are a lot more efficient – it's only to look at how tests fail > (versus time out). > But the biggest factor by far is the coverage that python upgrade dtests > have. > > It's been questioned whether they break with legit failures. They do. > There's way too much weird shit between versions, pre-commit testing has to > be our best tactic here, especially as we get more committers. > > Also failures can be flakies that take multiple builds before a release to be > noticed. And our CI system is already so saturated that our post-commit > builds are not that far off weekly builds anyway (i.e. nightly/weekly > schedule wouldn't save us anything, probably just add more load). > > > >> … I'm curious if the community sees a major flaw with a proposal like the >> following: >> We formally support 3 releases at a time >> We only release MAJOR (i.e. semver major). No more "5.0, 5.1, 5.2", would >> now be "5.0, 6.0, 7.0" >> We test and support online upgrades between supported releases >> Any removal or API breakage follows a "deprecate-then-release" cycle >> We cut a release every 12 months >> Implications for operators: >> Upgrade paths for online upgrades are simple and clear. T-2. >> "Forced" update cadence to stay on supported versions is 3 years >> If you adopt v1.0 it will be supported until v4.0 comes out 36 months later >> This gives users the flexibility to prioritize functionality vs. stability >> and to balance release validation costs >> Deprecation cycles are clear as are compatibility paths. >> Release timelines and feature availability are predictable and clear >> Implications for developers on the project: >> Support requirements for online upgrades are clear >> Opportunity cost of feature slippage relative to release date is balanced >> (worst-case == 11.99 month delay on availability in GA supported release) >> Path to keep code-base maintainable is clear (deprecate-then-remove) >> CI requirements are constrained and predictable > > > > FTR this^ implements my concerns stated in yesterday's post. > > > Small concerns are… > > - We need/should document the change (e.g. before X we did this, after X we > did that…). I like that it's a proposal that avoids any need for > documentation around arbitrary major versions – which no one has offered to > take on). > > - The T-2 is fixed, where using major versions gives us more flexibility. I > don't object to that at all, but it's worth highlighting. > > - When would T-2 kick in ? Would 6.0 mean everything from 4.0 (despite that > kinda being like a one-off T-3) ? > > >
