Thank you for your reply and good to know about the debug statement. I haven't
We never dropped or re-created the keyspace before. We haven't even performed writes to that keyspace in months. I also checked the permissions of Apache, that user had read only access. Unfortunately, I reverted from a backend recently. I cannot say for sure anymore if I saw something in system before the revert. Anyway, hopefully it was just a fluke. We have some crazy ML libraries running on it maybe Cassandra just gave up? Ohh well, Cassandra is a a champ and we haven't really had issues with it before. On Thu, Mar 2, 2017 at 6:51 PM, Romain Hardouin <romainh...@yahoo.fr> wrote: > Did you inspect system tables to see if there is some traces of your > keyspace? Did you ever drop and re-create this keyspace before that? > > Lines in debug appear because fd interval is > 2 seconds (logs are in > nanoseconds). You can override intervals via -Dcassandra.fd_initial_value_ms > and -Dcassandra.fd_max_interval_ms properties. Are you sure you didn't have > these lines in debug logs before? I used to see them a lot prior to > increase intervals to 4 seconds. > > Best, > > Romain > > Le Mardi 28 février 2017 18h25, George Webster <webste...@gmail.com> a > écrit : > > > Hey Cassandra Users, > > We recently encountered an issue with a keyspace just disappeared. I was > curious if anyone has had this occur before and can provide some insight. > > We are using cassandra 3.10. 2 DCs 3 nodes each. > The data was still located in the storage folder but is not located inside > Cassandra > > I searched the logs for any hints of error or commands being executed that > could have caused a loss of a keyspace. Unfortunately I found nothing. In > the logs the only unusual issue i saw was a series of read timeouts that > occurred right around when the keyspace went away. Since then I see > numerous entries in debug log as the following: > > DEBUG [GossipStage:1] 2017-02-28 18:14:12,580 FailureDetector.java:457 - > Ignoring interval time of 2155674599 <(215)%20567-4599> for /x.x.x..12 > DEBUG [GossipStage:1] 2017-02-28 18:14:16,580 FailureDetector.java:457 - > Ignoring interval time of 2945213745 for /x.x.x.81 > DEBUG [GossipStage:1] 2017-02-28 18:14:19,590 FailureDetector.java:457 - > Ignoring interval time of 2006530862 for /x.x.x..69 > DEBUG [GossipStage:1] 2017-02-28 18:14:27,434 FailureDetector.java:457 - > Ignoring interval time of 3441841231 for /x.x.x.82 > DEBUG [GossipStage:1] 2017-02-28 18:14:29,588 FailureDetector.java:457 - > Ignoring interval time of 2153964846 <(215)%20396-4846> for /x.x.x.82 > DEBUG [GossipStage:1] 2017-02-28 18:14:33,582 FailureDetector.java:457 - > Ignoring interval time of 2588593281 for /x.x.x.82 > DEBUG [GossipStage:1] 2017-02-28 18:14:37,588 FailureDetector.java:457 - > Ignoring interval time of 2005305693 for /x.x.x.69 > DEBUG [GossipStage:1] 2017-02-28 18:14:38,592 FailureDetector.java:457 - > Ignoring interval time of 2009244850 for /x.x.x.82 > DEBUG [GossipStage:1] 2017-02-28 18:14:43,584 FailureDetector.java:457 - > Ignoring interval time of 2149192677 <(214)%20919-2677> for /x.x.x.69 > DEBUG [GossipStage:1] 2017-02-28 18:14:45,605 FailureDetector.java:457 - > Ignoring interval time of 2021180918 for /x.x.x.85 > DEBUG [GossipStage:1] 2017-02-28 18:14:46,432 FailureDetector.java:457 - > Ignoring interval time of 2436026101 for /x.x.x.81 > DEBUG [GossipStage:1] 2017-02-28 18:14:46,432 FailureDetector.java:457 - > Ignoring interval time of 2436187894 for /x.x.x.82 > > During the time of the disappearing keyspace we had two concurrent > activities: > 1) Running a Spark job (via HDP 2.5.3 in Yarn) that was performing a > countbykey. It was using they Keyspace that disappeared. The operation > crashed. > 2) We created a new keyspace to test out scheme. Only "fancy" thing in > that keyspace are a few material view tables. Data was being loaded into > that keyspace during the crash. The load process was extracting information > and then just writing to Cassandra. > > Any ideas? Anyone seen this before? > > Thanks, > George > > >
