You ran the `alter keyspace` command on the original dc1 nodes or the new dc2 nodes?
On Wed, Oct 13, 2021 at 8:15 AM Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > Hi Tom, > > while I am not completely sure what might cause your issue, I just > want to highlight that schema agreements were overhauled in 4.0 (1) a > lot so that may be somehow related to what that ticket was trying to > fix. > > Regards > > (1) https://issues.apache.org/jira/browse/CASSANDRA-15158 > > On Fri, 1 Oct 2021 at 18:43, Tom Offermann <tofferm...@newrelic.com> > wrote: > > > > When adding a datacenter to a keyspace (following the Last Pickle [Data > Center Switch][lp] playbook), I ran into a "Configuration exception merging > remote schema" error. The nodes in one datacenter didn't converge to the > new schema version, and after restarting them, I saw the symptoms described > in this Datastax article on [Fixing a table schema collision][ds], where > there were two data directories for each table in the keyspace on the nodes > that didn't converge. I followed the recovery steps in the Datastax article > to move the data from the older directories to the new directories, ran > `nodetool refresh`, and that fixed the problem. > > > > [lp]: https://thelastpickle.com/blog/2019/02/26/data-center-switch.html > > [ds]: > https://docs.datastax.com/en/dse/6.0/cql/cql/cql_using/useCreateTableCollisionFix.html > > > > While the Datastax article was super helpful for helping me recover, I'm > left wondering *why* this happened. If anyone can shed some light on that, > or offer advice on how I can avoid getting in this situation in the future, > I would be most appreciative. I'll describe the steps I took in more detail > in the thread. > > > > ## Steps > > > > 1. The day before, I had added the second datacenter ('dc2') to the > system_traces, system_distributed, and system_auth keyspaces and ran > `nodetool rebuild` for each of the 3 keyspaces. All of that went smoothly > with no issues. > > > > 2. For a large keyspace, I added the second datacenter ('dc2') with an > `ALTER KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy', > 'dc1': '2', 'dc2': '3'};` statement. Immediately, I saw this error in the > log: > > ``` > > "ERROR 16:45:47 Exception in thread Thread[MigrationStage:1,5,main]" > > "org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 8ad72660-f629-11eb-a217-e1a09d8bc60c; expected > 20739eb0-d92e-11e6-b42f-e7eb6f21c481)" > > "\tat > org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:949) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:903) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.config.Schema.updateTable(Schema.java:687) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1482) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1438) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1407) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1384) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:594) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_232]" > > "\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_232]" > > "\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[na:1.8.0_232]" > > "\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_232]" > > "\tat > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > [apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]" > > ``` > > > > I also saw this: > > ``` > > "ERROR 16:46:48 Configuration exception merging remote schema" > > "org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 8ad72660-f629-11eb-a217-e1a09d8bc60c; expected > 20739eb0-d92e-11e6-b42f-e7eb6f21c481)" > > "\tat > org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:949) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:903) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.config.Schema.updateTable(Schema.java:687) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1482) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1438) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1407) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1384) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:91) > ~[apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > [apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > [apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_232]" > > "\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_232]" > > "\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_232]" > > "\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_232]" > > "\tat > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > [apache-cassandra-3.11.5.jar:3.11.5]" > > "\tat java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]" > > ``` > > This error repeated several times over the next 2 minutes. > > > > 3. While running `nodetool describecluster` repeatedly, I saw that the > nodes in the 'dc2' datacenter converged to the new schema version quickly, > but the nodes in the original datacenter ('dc1') remained at the previous > schema version. > > > > 4. I waited to see if all of the nodes would converge to the new schema > version, but they still hadn't converged after roughly 10 minutes. Given > the errors I saw, I wasn't optimistic it would work out all by itself, so I > decided to restart the nodes in the 'dc1' datacenter one at a time so they > would restart with the latest schema version. > > > > 5. After each node restarted, `nodetool describecluster` showed it as > being on the latest schema version. So, after getting through all the 'dc1' > nodes, it looked like everything in the cluster was healthy again. > > > > 6. However, that's when I noticed that there were two data directories > on disk for each table in the keyspace. New writes for a table were being > saved in the newer directory, but queries for data saved in the old data > directory were returning no results. > > > > 7. That's when I followed the recovery steps in the Datastax article > with great success. > > > > ## Questions > > > > * My understanding is that running concurrent schema updates should > always be avoided, since that can result in schema collisions. But, in this > case, I wasn't performing multiple schema updates. I was just running a > single `ALTER KEYSPACE` statement. Any idea why a single schema update > would result in a schema collision and two data directories per table? > > > > * Should I have waited longer before restarting nodes? Perhaps, given > enough time, the Cassandra nodes would have all converged on the correct > schema version, and this would have resolved on it's own? > > > > * Any suggestions for how I can avoid this problem in the future? > > > > -- > > Tom Offermann > > Lead Software Engineer > > http://newrelic.com >
