Ouch, all nodes are affected? That's a bit unusual...
If you don't have any useful data in the table, try drop the table and
then manually delete the data folder on all nodes. TBH I'm not even sure
if this is going to work, but worth to give it a try as it's simple enough.
Alternatively, if that didn't work, try shut down a node, remove the
data folder and restart the node, repeat this for all nodes and then
finally drop and re-create the table. If you do this, please run "DROP
TABLE" and "CREATE TABLE" against the same node.
To prevent this from happening again in the future, you can do one of
the following two things:
* Ensure schema changing statements are always sent to the same node; or
* Ensure at most one thread can issue schema changing statements at
anytime, and ensure it always check and wait for schema agreement on
all nodes before attempting to making any change.
The first option is usually the simpler option. Your normal application
code should not make schema changes, only the specific migration code
does that, and you should always run the migration code against a
specific Cassandra node. If that node has failed or is removed from the
cluster, manually check the schema agreement between nodes, and choose a
different node to use for all future schema changes. Don't do the
failover automatically as this can lead to racing conditions when the
node is flapping up and down.
On 25/01/2022 15:02, Amandeep Srivastava wrote:
Hi,
I tried looking for the Id we got from schema table in all 18 nodes of
my cluster (we maintain 9+9 hot/warm cluster), but none of the node
seems to have the same Id in its data folder.
Is there a mechanism in cass which would've cleaned the
duplicate/redundant id table?
To summarise, we have one Id mapped in the schema table and another Id
that is present in /data/ks of all 18 nodes. Is there a way to make
schema table Id point to correct one? (Would repair help here since
call nodes have same data?)
Regards,
Aman
On Tue, 25 Jan, 2022, 7:39 pm Amandeep Srivastava,
<amandeep.srivastava1...@gmail.com> wrote:
Ah, was expecting concurrency to be the cause. Lucky, we caught it
before we started ingesting any data in cassandra :)
Thanks for your help, Bowen, let me try this.
Regards,
Aman
On Tue, 25 Jan, 2022, 7:14 pm Bowen Song, <bo...@bso.ng> wrote:
The usual cause of this is the CQL statement CREATE TABLE ...
and/or
CREATE TABLE ... IF NOT EXISTS were ran on two nodes in a
short period
of time (i.e. before the schema is fully synced between the
nodes). If
both tables (same name, different UUIDs) have data in them,
fixing the
issue without losing data is a bit trickier. However, if any
of them is
empty, you should be able to shutdown the node with the empty
table,
delete the table's data folder on that node and then restart
the node,
repeat this for all nodes with the empty table, and finally
run a full
repair.
On 25/01/2022 07:32, Amandeep Srivastava wrote:
> Hi,
>
> We're running an embedded Janus graph on top of Cassandra.
On starting
> the graph, it creates certain tables in cassandra for its
operation.
> We noticed that there is an Id mismatch for one of the
tables named
> system_properties_lock i.e.
>
> Id fetched from schema table of cass (SELECT keyspace_name,
> table_name, id from system_schema.tables using cqlsh), and
the path in
> cassandra
(/opt/Cassandra/data/Janus/system-properties_lock_-id), do
> not match. IDs match for all other tables created by Janus
except this
> one.
>
> Can someone please advice on what might be the cause of this
mismatch
> and how can it be fixed?
>
> Regards,
> Aman
>
>
