I've been struggling with these kinds of exceptions for some time now. I
thought it might have been a one-time thing, so on the 2 nodes where I saw this
problem I pulled in fresh data with a repair on an empty data directory.
Unfortunately, this problem is now coming up on a new node that has, up until
now, not had this problem.
What could be causing this? Could it be related to encoding? Why are these
rows not readable?
This exception prevents cassandra from doing repairs, and even minor
compactions. It also messes up memtable management (with a normal load of
25GB, disk goes to almost 100% full on a 500 GB hd).
This is incredibly frustrating. This is the only pain-point I have had with
cassandra so far. By the way, this node was never upgraded - it was 0.7.4
from the start, so that eliminates format compatibility problems.
ERROR [CompactionExecutor:1] 2011-04-15 21:31:23,479 PrecompactedRow.java (line
82) Skipping row DecoratedKey(105452551814086725777389040553659117532,
4d657373616765456e726963686d656e743a313032343937) in
/var/lib/cassandra/data/DFS/main-f-91-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:147)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
