I have two tables. Table “event” stores data with
SizeTieredCompactionStrategy, and table “event_index” acts as an index table
with LeveledCompationStrategy, and with TimeUUID as its clustering column.
For each record in event table, there will be 4 indices in the event_index
table, and the overall size of the 4 indices is about the same as each
record in event table. All columns have TTL set so the record and the
corresponding indices go away at the same time.
About a week ago, I load tested it for 3 days with millions of records. The
majority of the records have 7 day TTL, while a small amount of records have
30 day TTL.
Now a simple CQL query like “select * from event_index limit 1” won’t run
and Cassandra log says
ERROR [ReadStage:68] 2014-10-01 15:40:14,751 SliceQueryFilter.java (line
200) Scanned over 100000 tombstones in event_index; query aborted (see
tombstone_fail_threshold)
ERROR [ReadStage:68] 2014-10-01 15:40:14,753 CassandraDaemon.java (line 196)
Exception in thread Thread[ReadStage:68,5,main]
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1900)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
I can understand why there could be that many tombstones, but should they be
removed by compaction?
I went to the corresponding keyspace folder. To make sure I only count
actual table files, I delete everything under tablename/backups and
tablename/snapshots folders. Then in the keyspace folder, I ran “du” command
to estimate the file space usage.
$ du -sk *
239872 event
248092 event_index
Then I ran “nodetool compact” against the keyspace, and ran “du” again
$ du -sk *
80048 event
248092 event_index
Well, while event get compacted and its size dropped to 1/3 of the original,
event_index did not budge. I remember from reading somewhere that “nodetool
compact” is a no-op for LeveledCompationStrategy tables, which explains why
event_index size did not change.
I then altered the table with gc_grace_seconds=600 and hoped it will speed
up the compaction. However, it has been two days and I don’t see any change.
What can I do at this point to get the event_index table compacted and get
those tombstones removed?
Another question, should I not use LeveledCompationStrategy for this type of
time series table?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/CQL-query-throws-TombstoneOverwhelmingException-against-a-LeveledCompactionStrategy-table-tp7597077.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
