[
https://issues.apache.org/jira/browse/CASSANDRA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975853#action_12975853
]
Gary Dusbabek commented on CASSANDRA-1912:
------------------------------------------
Current behavior grabs the flush lock, force a flush, drop the CF, then
releases the flush lock. So a flush *is* happening. Also, CL recovery
actively checks for an handles mutations where the column family is no longer
present (deserialization fails and recovery goes on to the next record).
I tried reproducing in my environment following these steps (but couldn't see
the problem):
1. spin up a node, load schema.
2. make 5k inserts.
3. drop the keyspace,
4. kill cassandra, restart.
Can you try it again but with DEBUG turned on? You should see lot of lines
like "Reading mutation at <long>".
> Cassandra should flush a keyspace after it was dropped
> ------------------------------------------------------
>
> Key: CASSANDRA-1912
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1912
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.0 rc 2
> Environment: any
> Reporter: Daniel Kraft
> Assignee: Gary Dusbabek
> Fix For: 0.7.1
>
>
> After dropping a keyspace one must not forget to flush it via
> nodetool -h host flush <dropped keyspace>
> If you forget to do this and restart cassandra, the following traceback
> happens and cassandra doesn't start up again:
> INFO 20:22:56,420 Heap size: 4117889024/4137549824
> INFO 20:22:56,424 JNA not found. Native methods will be disabled.
> INFO 20:22:56,432 Loading settings from
> file:/home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/conf/cassandra.yaml
> INFO 20:22:56,547 DiskAccessMode 'auto' determined to be mmap,
> indexAccessMode is mmap
> INFO 20:22:56,612 Creating new commitlog segment
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/commitlog/CommitLog-1293564176612.log
> INFO 20:22:56,666 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/system-IndexInfo-KeyCache
> INFO 20:22:56,673 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/IndexInfo-e-1
> INFO 20:22:56,698 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/system-Schema-KeyCache
> INFO 20:22:56,700 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/Schema-e-41
> INFO 20:22:56,703 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/Schema-e-43
> INFO 20:22:56,705 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/Schema-e-42
> INFO 20:22:56,727 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/system-Migrations-KeyCache
> INFO 20:22:56,728 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/Migrations-e-41
> INFO 20:22:56,730 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/Migrations-e-42
> INFO 20:22:56,734 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/system-LocationInfo-KeyCache
> INFO 20:22:56,735 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/LocationInfo-e-6
> INFO 20:22:56,739 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/LocationInfo-e-7
> INFO 20:22:56,741 Opening
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/data/system/LocationInfo-e-5
> INFO 20:22:56,746 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/system-HintsColumnFamily-KeyCache
> INFO 20:22:56,776 Loading schema version 460e882a-1291-11e0-92ff-e700f669bcfc
> WARN 20:22:56,943 Schema definitions were defined both locally and in
> cassandra.yaml. Definitions in cassandra.yaml were ignored.
> INFO 20:22:56,954 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Indexed1-KeyCache
> INFO 20:22:56,959 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Super1-KeyCache
> INFO 20:22:56,960 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Standard2-KeyCache
> INFO 20:22:56,961 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Super2-KeyCache
> INFO 20:22:56,962 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Standard1-KeyCache
> INFO 20:22:56,963 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-Super3-KeyCache
> INFO 20:22:56,964 reading saved cache
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/saved_caches/Keyspace1-StandardByUUID1-KeyCache
> INFO 20:22:56,971 Replaying
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/commitlog/CommitLog-1293524239077.log,
>
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/commitlog/CommitLog-1293563866498.log
> INFO 20:22:56,974 Finished reading
> /home/dk/develop/cassandra/apache-cassandra-0.7.0-rc3/STORAGE/commitlog/CommitLog-1293524239077.log
> ERROR 20:22:56,975 Exception encountered during startup.
> java.lang.NullPointerException
> at
> org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:318)
> at org.apache.cassandra.db.Table.<init>(Table.java:258)
> at org.apache.cassandra.db.Table.open(Table.java:107)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:302)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:193)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:142)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)
> Exception encountered during startup.
> java.lang.NullPointerException
> at
> org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:318)
> at org.apache.cassandra.db.Table.<init>(Table.java:258)
> at org.apache.cassandra.db.Table.open(Table.java:107)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:302)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:193)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:142)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)
> On IRC the only hint was to move the commitlog away, start cassandra, create
> the keyspace, stop cassandra, move the commitlog back and start it again.
> I'm not sure, how stable this would be in a loaded environment, so I propose
> that cassandra does the flush itself after dropping a keyspace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
