Hi list,

We run Cassandra 1.2 on three-node cluster. Each node has 16GB RAM, single
200GB HDD with Ubuntu Server 12.04.

There is an issue with one table that contains about 3000 rows, here its
describe-table:

CREATE TABLE outputs (
  appid text,
  staged boolean,
  field ascii,
  data blob,
  PRIMARY KEY (appid, staged, field)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Column DATA contains blobs of size about 1..50MB, average size should be
something of 5MB.

Sometimes this table expiriences huge write loads for few hours, at such
times I see suspicious things in logs:

ERROR [CompactionExecutor:357] 2013-07-24 12:32:10,293 CassandraDaemon.java
(line 192) Exception in thread Thread[CompactionExecutor:357,1,main]
java.lang.AssertionError: incorrect row data size 172489604 written to
/var/lib/cassandra/data/woodpecker/outputs/woodpecker-outputs-tmp-ic-813-Data.db;
correct is 172489704
        at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
        at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
        at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
        at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
        at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)

What shall I do about this?

Thanks in advance.
Pavel.

Reply via email to