> I think it would be a worse experience to not warn and let the user discover later when they can't write at QUORUM.
Agree. Should we add a note in the cassandra.yaml comments as well? Maybe in the spot where default_keyspace_rf is defined? On the other hand, that section is pretty "wordy" already. But calling it out in the yaml might not be a bad idea. Thanks, Aaron On Tue, Mar 7, 2023 at 11:12 AM Derek Chen-Becker <de...@chen-becker.org> wrote: > I think that the warning would only be thrown in the case where a > potentially QUORUM-busting configuration is used. I think it would be a > worse experience to not warn and let the user discover later when they > can't write at QUORUM. > > Cheers, > > Derek > > On Tue, Mar 7, 2023 at 9:32 AM Jeremiah D Jordan < > jeremiah.jor...@gmail.com> wrote: > >> I agree with Paulo, it would be nice if we could figure out some way to >> make new NTS work correctly, with a parameter to fall back to the “bad” >> behavior, so that people restoring backups to a new cluster can get the >> right behavior to match their backups. >> The problem with only fixing this in a new strategy is we have a ton of >> tutorials and docs out there which tell people to use NTS, so it would be >> great if we could keep “use NTS” as the recommendation. Throwing a warning >> when someone uses NTS is kind of user hostile. If someone just read some >> tutorial or doc which told them “make your key space this way” and then >> when they do that the database yells at them telling them they did it >> wrong, it is not a great experience. >> >> -Jeremiah >> >> > On Mar 7, 2023, at 10:16 AM, Benedict <bened...@apache.org> wrote: >> > >> > My view is that if this is a pretty serious bug. I wonder if >> transactional metadata will make it possible to safely fix this for users >> without rebuilding (only via opt-in, of course). >> > >> >> On 7 Mar 2023, at 15:54, Miklosovic, Stefan < >> stefan.mikloso...@netapp.com> wrote: >> >> >> >> Thanks everybody for the feedback. >> >> >> >> I think that emitting a warning upon keyspace creation (and >> alteration) should be enough for starters. If somebody can not live without >> 100% bullet proof solution over time we might choose some approach from the >> offered ones. As the saying goes there is no silver bullet. If we decide to >> implement that new strategy, we would probably emit warnings anyway on NTS >> but it would be already done so just new strategy would be provided. >> >> >> >> ________________________________________ >> >> From: Paulo Motta <pauloricard...@gmail.com> >> >> Sent: Monday, March 6, 2023 17:48 >> >> To: dev@cassandra.apache.org >> >> Subject: Re: Degradation of availability when using NTS and RF > >> number of racks >> >> >> >> NetApp Security WARNING: This is an external email. Do not click links >> or open attachments unless you recognize the sender and know the content is >> safe. >> >> >> >> >> >> >> >> It's a bit unfortunate that NTS does not maintain the ability to lose >> a rack without loss of quorum for RF > #racks > 2, since this can be easily >> achieved by evenly placing replicas across all racks. >> >> >> >> Since RackAwareTopologyStrategy is a superset of >> NetworkTopologyStrategy, can't we just use the new correct placement logic >> for newly created keyspaces instead of having a new strategy? >> >> >> >> The placement logic would be backwards-compatible for RF <= #racks. On >> upgrade, we could mark existing keyspaces with RF > #racks with >> use_legacy_replica_placement=true to maintain backwards compatibility and >> log a warning that the rack loss guarantee is not maintained for keyspaces >> created before the fix. Old keyspaces with RF <=#racks would still work >> with the new replica placement. The downside is that we would need to keep >> the old NTS logic around, or we could eventually deprecate it and require >> users to migrate keyspaces using the legacy placement strategy. >> >> >> >> Alternatively we could have RackAwareTopologyStrategy and fail NTS >> keyspace creation for RF > #racks and indicate users to use >> RackAwareTopologyStrategy to maintain the quorum guarantee on rack loss or >> set an override flag "support_quorum_on_rack_loss=false". This feels a bit >> iffy though since it could potentially confuse users about when to use each >> strategy. >> >> >> >> On Mon, Mar 6, 2023 at 5:51 AM Miklosovic, Stefan < >> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote: >> >> Hi all, >> >> >> >> some time ago we identified an issue with NetworkTopologyStrategy. The >> problem is that when RF > number of racks, it may happen that NTS places >> replicas in such a way that when whole rack is lost, we lose QUORUM and >> data are not available anymore if QUORUM CL is used. >> >> >> >> To illustrate this problem, lets have this setup: >> >> >> >> 9 nodes in 1 DC, 3 racks, 3 nodes per rack. RF = 5. Then, NTS could >> place replicas like this: 3 replicas in rack1, 1 replica in rack2, 1 >> replica in rack3. Hence, when rack1 is lost, we do not have QUORUM. >> >> >> >> It seems to us that there is already some logic around this scenario >> (1) but the implementation is not entirely correct. This solution is not >> computing the replica placement correctly so the above problem would be >> addressed. >> >> >> >> We created a draft here (2, 3) which fixes it. >> >> >> >> There is also a test which simulates this scenario. When I assign 256 >> tokens to each node randomly (by same mean as generatetokens command uses) >> and I try to compute natural replicas for 1 billion random tokens and I >> compute how many cases there will be when 3 replicas out of 5 are inserted >> in the same rack (so by losing it we would lose quorum), for above setup I >> get around 6%. >> >> >> >> For 12 nodes, 3 racks, 4 nodes per rack, rf = 5, this happens in 10% >> cases. >> >> >> >> To interpret this number, it basically means that with such topology, >> RF and CL, when a random rack fails completely, when doing a random read, >> there is 6% chance that data will not be available (or 10%, respectively). >> >> >> >> One caveat here is that NTS is not compatible with this new strategy >> anymore because it will place replicas differently. So I guess that fixing >> this in NTS will not be possible because of upgrades. I think people would >> need to setup completely new keyspace and somehow migrate data if they wish >> or they just start from scratch with this strategy. >> >> >> >> Questions: >> >> >> >> 1) do you think this is meaningful to fix and it might end up in trunk? >> >> >> >> 2) should not we just ban this scenario entirely? It might be possible >> to check the configuration upon keyspace creation (rf > num of racks) and >> if we see this is problematic we would just fail that query? Guardrail >> maybe? >> >> >> >> 3) people in the ticket mention writing "CEP" for this but I do not >> see any reason to do so. It is just a strategy as any other. What would >> that CEP would even be about? Is this necessary? >> >> >> >> Regards >> >> >> >> (1) >> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java#L126-L128 >> >> (2) https://github.com/apache/cassandra/pull/2191 >> >> (3) https://issues.apache.org/jira/browse/CASSANDRA-16203 >> > >> >> > > -- > +---------------------------------------------------------------+ > | Derek Chen-Becker | > | GPG Key available at https://keybase.io/dchenbecker and | > | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | > | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | > +---------------------------------------------------------------+ > >
