Re: question of keyspace that just disappeared

2017-03-03 Thread George Webster
I think it does on drop keyspace. We had a recent enough snapshot so it wasn't a big deal to recover. However, we didn't have a snapshot for when the keyspace disappeared. @Romain: I believe you are correct about reliability. We just had a repair --full fail and CPU lock up one of the nodes at 100

Re: question of keyspace that just disappeared

2017-03-03 Thread Edward Capriolo
On Fri, Mar 3, 2017 at 7:56 AM, Romain Hardouin wrote: > I suspect a lack of 3.x reliability. Cassandra could had gave up with > dropped messages but not with a "drop keyspace". I mean I already saw some > spark jobs with too much executors that produce a high load average on a > DC. I saw a C* n

Re: question of keyspace that just disappeared

2017-03-03 Thread Romain Hardouin
I suspect a lack of 3.x reliability. Cassandra could had gave up with dropped messages but not with a "drop keyspace". I mean I already saw some spark jobs with too much executors that produce a high load average on a DC. I saw a C* node with a 1 min. load avg of 140 that can still have a P99 re

Re: question of keyspace that just disappeared

2017-03-03 Thread George Webster
Thank you for your reply and good to know about the debug statement. I haven't We never dropped or re-created the keyspace before. We haven't even performed writes to that keyspace in months. I also checked the permissions of Apache, that user had read only access. Unfortunately, I reverted from

Re: question of keyspace that just disappeared

2017-03-02 Thread Romain Hardouin
Did you inspect system tables to see if there is some traces of your keyspace? Did you ever drop and re-create this keyspace before that? Lines in debug appear because fd interval is > 2 seconds (logs are in nanoseconds). You can override intervals via -Dcassandra.fd_initial_value_ms and -Dcassa

question of keyspace that just disappeared

2017-02-28 Thread George Webster
Hey Cassandra Users, We recently encountered an issue with a keyspace just disappeared. I was curious if anyone has had this occur before and can provide some insight. We are using cassandra 3.10. 2 DCs 3 nodes each. The data was still located in the storage folder but is not located inside Cass