Re: MISSING keyspace

2021-03-01 Thread Marco Gasparini
hi @Erick, Actually this timestamp *1614575293790 *is equivalent to *GMT: Monday, 1 March 2021 05:08:13.790* that stands for *GMT+1: Monday, 1 March 2021 06:08:13.790* (my local timezone). This is consistent with the other logs time in the cluster. Thank yo

Re: MISSING keyspace

2021-03-01 Thread Erick Ramirez
The timestamp (1614575293790) in the snapshot directory name is equivalent to 1 March 16:08 GMT: actually I found a lot of .db files in the following directory: > > /var/lib/cassandra/data/mykespace/mytable-2795c0204a2d11e9aba361828766468f/snapshots/dropped-1614575293790- > mytable > which lines

Re: MISSING keyspace

2021-03-01 Thread Marco Gasparini
I haven't made any schema modifications for a year or more. This problem came up during a "normal day of work" for Cassandra. Il giorno lun 1 mar 2021 alle ore 16:25 Bowen Song ha scritto: > Your missing keyspace problem has nothing to do with that bug. > > In that case, the same table was cre

Re: MISSING keyspace

2021-03-01 Thread Bowen Song
Your missing keyspace problem has nothing to do with that bug. In that case, the same table was created twice in a very short period of time, and I suspect that was done concurrently on two different nodes. The evidence lies in the two CF IDs - bd7200a0156711e88974855d74ee356f and bd750de01567

Re: MISSING keyspace

2021-03-01 Thread Marco Gasparini
actually I found a lot of .db files in the following directory: /var/lib/cassandra/data/mykespace/mytable-2795c0204a2d11e9aba361828766468f/snapshots/dropped-1614575293790- mytable I also found this: 2021-03-01 06:08:08,864 INFO [Native-Transport-Requests-1] MigrationManager.java:542

Re: MISSING keyspace

2021-03-01 Thread Bowen Song
The warning message indicates the node y.y.y.y went down (or is unreachable via network) before 2021-02-28 05:17:33. Is there any chance you can find the log file on that node at around or before that time? It may show why did that node go down. The reason of that might be irrelevant to the mis

Re: MISSING keyspace

2021-03-01 Thread Erick Ramirez
As the warning message suggests, you need to check for schema disagreement. My suspicion is that someone made a schema change and possibly dropped the problematic keyspace. FWIW I suspect the keyspace was dropped because the table isn't new -- CF ID cba90a70-5c46-11e9-9e36-f54fe3235e69 is equivale

Re: MISSING keyspace

2021-03-01 Thread Marco Gasparini
here the previous error: 2021-02-28 05:17:33,262 WARN NodeConnectionsService.java:165 validateAndConnectIfNeeded failed to connect to node {y.y.y.y}{9ba2d3ee-bc82-4e76-ae24-9e20eb334c24}{9ba2d3ee-bc82-4e76-ae24-9e20eb334c24}{ y.y.y.y }{ y.y.y.y :9300}{ALIVE}{rack=r1, dc=DC1} (tried [1] times) org.

Re: MISSING keyspace

2021-03-01 Thread Bowen Song
What was the warning? Is it related to the disk failure policy? Could you please share the relevant log? You can edit it and redact the sensitive information before sharing it. Also, I can't help to notice that you used the word "delete" (instead of "clear") to describe the process of removing

Re: MISSING keyspace

2021-03-01 Thread Marco Gasparini
thanks Bowen for answering Actually, I checked the server log and the only warning was that a node went offline. No, I have no backups or snapshots. In the meantime I found that probably Cassandra moved all files from a directory to the snapshot directory. I am pretty sure of that because I have

Re: MISSING keyspace

2021-03-01 Thread Bowen Song
The first thing I'd check is the server log. The log may contain vital information about the cause of it, and that there may be different ways to recover from it depending on the cause. Also, please allow me to ask a seemingly obvious question, do you have a backup? On 01/03/2021 09:34, Mar