That is very helpful, thank you so much, Bowen. I dropped the keyspace and created a new one (like with a new name). Things look good now.
Will modify the service bootups to introduce the checks you've mentioned, and that should mitigate the issue for any new clusters we'd setup. Regards, Aman On Tue, 25 Jan, 2022, 9:14 pm Bowen Song, <bo...@bso.ng> wrote: > Ouch, all nodes are affected? That's a bit unusual... > > If you don't have any useful data in the table, try drop the table and > then manually delete the data folder on all nodes. TBH I'm not even sure if > this is going to work, but worth to give it a try as it's simple enough. > > Alternatively, if that didn't work, try shut down a node, remove the data > folder and restart the node, repeat this for all nodes and then finally > drop and re-create the table. If you do this, please run "DROP TABLE" and > "CREATE TABLE" against the same node. > > To prevent this from happening again in the future, you can do one of the > following two things: > > - Ensure schema changing statements are always sent to the same node; > or > - Ensure at most one thread can issue schema changing statements at > anytime, and ensure it always check and wait for schema agreement on all > nodes before attempting to making any change. > > The first option is usually the simpler option. Your normal application > code should not make schema changes, only the specific migration code does > that, and you should always run the migration code against a specific > Cassandra node. If that node has failed or is removed from the cluster, > manually check the schema agreement between nodes, and choose a different > node to use for all future schema changes. Don't do the failover > automatically as this can lead to racing conditions when the node is > flapping up and down. > > > > On 25/01/2022 15:02, Amandeep Srivastava wrote: > > Hi, > > I tried looking for the Id we got from schema table in all 18 nodes of my > cluster (we maintain 9+9 hot/warm cluster), but none of the node seems to > have the same Id in its data folder. > > Is there a mechanism in cass which would've cleaned the > duplicate/redundant id table? > > To summarise, we have one Id mapped in the schema table and another Id > that is present in /data/ks of all 18 nodes. Is there a way to make schema > table Id point to correct one? (Would repair help here since call nodes > have same data?) > > Regards, > Aman > > On Tue, 25 Jan, 2022, 7:39 pm Amandeep Srivastava, < > amandeep.srivastava1...@gmail.com> wrote: > >> Ah, was expecting concurrency to be the cause. Lucky, we caught it before >> we started ingesting any data in cassandra :) >> >> Thanks for your help, Bowen, let me try this. >> >> Regards, >> Aman >> >> >> On Tue, 25 Jan, 2022, 7:14 pm Bowen Song, <bo...@bso.ng> wrote: >> >>> The usual cause of this is the CQL statement CREATE TABLE ... and/or >>> CREATE TABLE ... IF NOT EXISTS were ran on two nodes in a short period >>> of time (i.e. before the schema is fully synced between the nodes). If >>> both tables (same name, different UUIDs) have data in them, fixing the >>> issue without losing data is a bit trickier. However, if any of them is >>> empty, you should be able to shutdown the node with the empty table, >>> delete the table's data folder on that node and then restart the node, >>> repeat this for all nodes with the empty table, and finally run a full >>> repair. >>> >>> >>> On 25/01/2022 07:32, Amandeep Srivastava wrote: >>> > Hi, >>> > >>> > We're running an embedded Janus graph on top of Cassandra. On starting >>> > the graph, it creates certain tables in cassandra for its operation. >>> > We noticed that there is an Id mismatch for one of the tables named >>> > system_properties_lock i.e. >>> > >>> > Id fetched from schema table of cass (SELECT keyspace_name, >>> > table_name, id from system_schema.tables using cqlsh), and the path in >>> > cassandra (/opt/Cassandra/data/Janus/system-properties_lock_-id), do >>> > not match. IDs match for all other tables created by Janus except this >>> > one. >>> > >>> > Can someone please advice on what might be the cause of this mismatch >>> > and how can it be fixed? >>> > >>> > Regards, >>> > Aman >>> > >>> > >>> >>
