Hi list,
We run Cassandra 1.2 on three-node cluster. Each node has 16GB RAM, single
200GB HDD with Ubuntu Server 12.04.
There is an issue with one table that contains about 3000 rows, here its
describe-table:
CREATE TABLE outputs (
appid text,
staged boolean,
field ascii,
data blob,
PRIMARY KEY (appid, staged, field)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
Column DATA contains blobs of size about 1..50MB, average size should be
something of 5MB.
Sometimes this table expiriences huge write loads for few hours, at such
times I see suspicious things in logs:
ERROR [CompactionExecutor:357] 2013-07-24 12:32:10,293 CassandraDaemon.java
(line 192) Exception in thread Thread[CompactionExecutor:357,1,main]
java.lang.AssertionError: incorrect row data size 172489604 written to
/var/lib/cassandra/data/woodpecker/outputs/woodpecker-outputs-tmp-ic-813-Data.db;
correct is 172489704
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
What shall I do about this?
Thanks in advance.
Pavel.