Re: Downgradability

Hello everyone : What is the final conclusion of this discuss ? As https://issues.apache.org/jira/browse/CASSANDRA-18934 has been created , and we know that for system table of different c* version , the schema info may be different as we may add or delete column or modify the table properties in

Re: Downgradability

A bit of organization - I've created https://issues.apache.org/jira/browse/CASSANDRA-18300 epic to track tickets related to the downgradability. Please add the tickets you are aware of. thanks - - -- --- - - Jacek Lewandowski czw., 23 lut 2023 o 17:47 Benedict napisał(a

Re: Downgradability

Either way, it feels like this has become much more of a big deal than it needed to.I would prefer the pending patches to avoid breaking compatibility, as I think they can do it easily. But, if we agree to block release until we can double back to fix it with versioned writing (which I agree with J

Re: Downgradability

Right. So I'm speculating everyone else who worked on a patch that breaks compatibility has been working under the mindset "I'll just put this behind the same switch". Or something more vague / even less correct, such as assuming that tries would become the default immediately. At least in my mind

Re: Downgradability

I don’t think there’s anything about a new format that requires a version bump, but I could be missing something.We have to have a switch to enable tries already don’t we? I’m pretty sure we haven’t talked about switching the default format?On 23 Feb 2023, at 12:12, Henrik Ingo wrote:On Thu, Feb

Re: Downgradability

On Thu, Feb 23, 2023 at 11:57 AM Benedict wrote: > Can somebody explain to me why this is being fought tooth and nail, when > the work involved is absolutely minimal? > > I don't know how each individual has been thinking about this, but it seems to me just looking at all the tasks that at least

Re: Downgradability

You also need to remove the system.local.broadcast_port column as that does not exist in the earlier version and when the earlier version attempts to read it the code throws an UnknownColumnException. On Thu, Feb 23, 2023 at 11:27 AM Jacek Lewandowski < lewandowski.ja...@gmail.com> wrote: > Runni

Re: Downgradability

Running upgrade tests backwards is great idea which does not require extra work. For stats metadata it already supports writing in previous serialization version We need a small fix in compression metadata and that's it. A flag with the write format version is probably LHF. Maybe let's try, we

Re: Downgradability

Forget downgradeability for a moment: we should not be breaking format compatibility without good reason. Bumping a major version isn’t enough of a reason. Can somebody explain to me why this is being fought tooth and nail, when the work involved is absolutely minimal?Regarding tests: what more do

Re: Downgradability

> > Can somebody explain to me what is so burdensome, that we seem to be > spending longer debating it than it would take to implement the necessary > changes? I believe that we all agree on the outcome. Everybody wants downgradability. The issue is on the path to get there. As far as I am conce

Re: Downgradability

Broken downgrading can be fixed (I think) by modifying the SearializationHeader.toHeader() method where it currently throws an UnknownColumnException. If we can, instead of throwing the exception, create a dropped column for the unexpected column then I think the code will work. I realise that to

Re: Downgradability

... ok apparently shift+enter sends messages now? I was just saying if at least the file format AND system/tables - anything written to disk - can be protected with a switch, then it allows for quick downgrade by shutting down the entire cluster and restarting with the downgraded binary. It's a s

Re: Downgradability

Just this once I'm going to be really brief :-) Just wanted to share for reference how Mongodb implemented downgradeability around their 4.4 version: https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/ Jeff you're right. Ultimately this is about more than file formats

Re: Downgradability

Those tickets mostly do not need to break compatibility, and it is pretty easy for them to avoid doing so without any additional facilities.Only the TTL ticket has an excuse, as it debatably needs to support a higher version under certain non-default config settings. However there are no serialisat

Re: Downgradability

We have multiple tickets about to merge that introduce new on disk format changes. I see no reason to block those indefinitely while we figure out how to do the on disk format downgrade stuff. -Jeremiah > On Feb 22, 2023, at 3:12 PM, Benedict wrote: > > Ok I will be honest, I was fairly sure

Re: Downgradability

Ok I will be honest, I was fairly sure we hadn’t yet broken downgrade - but I was wrong. CASSANDRA-18061 introduced a new column to a system table, which is a breaking change. But that’s it, as far as I can tell. I have run a downgrade test successfully after reverting that ticket, using the one li

Re: Downgradability

> why not implement backwards write compatibility? +1 to this from a philosophical perspective. Keeping prior releases completely in the dark about new release sstable formats is a clean approach, and we should already have the code around to ser/deser the prior version's data on the next versio

Re: Downgradability

When people are serious about this requirement, they’ll build the downgrade equivalents of the upgrade tests and run them automatically, often, so people understand what the real gap is and when something new makes it break Until those tests exist, I think collectively we should all stop pretending

Re: Downgradability

> 1. Major SSTable changes should begin with forward-compatibility in a prior release. This requires "feature" changes, i.e. new non-trivial code for previous patch releases. It also entails porting over any further format modification. Instead of this, in combination with your second point, why

Re: Downgradability

Some interesting existing work on this subject is "Understanding and Detecting Software Upgrade Failures in Distributed Systems" - https://dl.acm.org/doi/10.1145/3477132.3483577, also summarized by Andrey Satarin here: https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-sys

Re: Downgradability

If we can get opt-in major format upgrades, as well as an offline sstabledowngrade tool, I think we have a good first step that would make downgrades possible. Given Jacek’s work on the sstable format API, and the work from Yuki and Claude on old formats, I think we are pretty close to having b

Re: Downgradability

As always, Scott puts it much more eloquently than I can. The only thing I’d quibble with is that I think it is better to make changes backwards compatible, rather than make earlier releases forwards compatible - and where this is prohibitively costly to simply make a feature that depends on it una

Re: Downgradability

I realize my feedback on this has been spread across tickets and older mailing list / wiki discussions, so I'll offer a proposal here.Starting with goals -1. Cassandra users must be able to abort and revert an upgrade to a new version of the database that introduces a new major SSTable format.Thi

Re: Downgradability

My goal in implementing CASSANDRA-8928 was to be able to take the current version 4.x and write it as the earliest 3.x version possible. The reasoning being that if that was possible then whatever 3.x version was executed

Re: Downgradability

This isn’t a feature in any normal sense? It’s a commitment, and it requires every contributor to consider it as part of work they produce.I agree we should put in some work to make this easier, and to provide some test guarantees that we don’t break it. But are we overstating the difficulty? We al

Re: Downgradability

It appears to me that the first thing we need to start this feature off is a definition of a suite of tests together with a set of rules to keep the suite up to date with new features as they are introduced. The moment that suite is in place, we can start having some confidence that we can enforce

Re: Downgradability

> > It seems to me the real issue is nobody noticed this was agreed and/or > forgot and didn’t think about it much. I have some trouble keeping up with all the discussions those days so I might have missed the discussion. The only place I remembered where this subject was mentioned was during the

Re: Downgradability

I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce multiple sstable formats support. It allows for providing an implementation of SSTableFormat along with SSTableReader and SSTableWriter. That could be extended easily to support different implementations for certain version r

Re: Downgradability

Hi, What I wanted to address in my comment in CASSANDRA-8110( https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705) is to focus on better upgrade experience. Upgrading the cluster ca

Re: Downgradability

FWIW I think 8110 is the right approach, even if it isn’t a panacea. We will have to eventually also tackle system schema changes (probably not hard), and may have to think a little carefully about other things, eg with TTLs the format change is only the contract about what values can be present, s

Re: Downgradability

In a self-organising community, things that aren’t self-policed naturally end up policed in an adhoc manner, and with difficulty. I’m not sure that’s the same as arbitrary enforcement. It seems to me the real issue is nobody noticed this was agreed and/or forgot and didn’t think about it much. But,

Re: Downgradability

I'm not even convinced even 8110 addresses this - just writing sstables in old versions won't help if we ever add things like new types or new types of collections without other control abilities. Claude's other email in another thread a few hours ago talks about some of these surprises - "Specific

Re: Downgradability

I’m a big fan of maintaining backward compatibility. Downgradability implies that we could potentially roll back an upgrade at any time. While I don’t think we need to retain the ability to downgrade in perpetuity it would be a good objective to maintain strict backward compatibility and therefore

Re: Downgradability

There has been progress on https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928 Which is similar to what datastax does for DSE. Would this be an acceptable solution? Jake On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell wrote: > It seems “An alternative solution is to impleme

Re: Downgradability

It seems “An alternative solution is to implement/complete CASSANDRA-8110 ” can give us more options if it is finished😉 Branimir Lambov 于2023年2月20日 周一下午11:03写道： > Hi everyone, > > There has been a discussion lately about changes to the sstable