Sadly repair isn't very resilient to errors and has failed. There is a few ticket open to improve this and repair in general but right now, if any problems occurs during repairs, it will fail (and nodetool repair won't return, so you could just ctrl-c).
Provided you're on a recent enough cassandra, I suggest you run scrub on the node giving the error. -- Sylvain On Fri, Apr 8, 2011 at 9:19 AM, Jonathan Colby <jonathan.co...@gmail.com> wrote: > It seems on my cluster there are a few unserializable Rows. I'm trying to > run a repair on the nodes, but it also seems that the replica nodes have > unreadable or unserializable rows. The problem is, I cannot determine if > the repair is still going on, or if was interrupted because of these errors. > It is unclear because nothing else related to the repair show up in the > logs. It's been about 5 hours and I also don't see anything happening when I > perform a "nodetool netstats" on the nodes. The nodetool repair command is > still blocking from the console. > > On the node I'm trying to repair, I see this after launching a "repair": > > ... > INFO [manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc] 2011-04-08 > 11:41:55,520 AntiEntropyService.java (line 770) Waiting for repair requests: > [#<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-7 > 9caafd7d3cc, /10.46.108.102, (DFS,main)>, #<TreeRequest > manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc, /10.46.108.101, > (DFS,main)>, #<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc > , /10.46.108.100, (DFS,main)>, #<TreeRequest > manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc, /10.47.108.101, > (DFS,main)>] > ... > > In the log of the node "10.46.108.102" where the repair tries to compare the > replica data, I see a couple of the below exceptions a few minutes later. > Are the exceptions bad enough to cause the repair to fail? > > > ERROR [CompactionExecutor:1] 2011-04-08 11:43:01,177 PrecompactedRow.java > (line 82) Skipping row DecoratedKey(1782314446006375058060694305099335169, > 4d657373616765456e726963686d656e743a31343236) in /va > r/lib/cassandra/data/DFS/main-f-177-Data.db > java.io.EOFException > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176) > at > org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78) > at > org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139) > at > org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108) > at > org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43) > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at > org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) > at > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) > at > org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803) > at > org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56) > at > org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [CompactionExecutor:1] 2011-04-08 11:43:53,762 PrecompactedRow.java > (line 82) Skipping row DecoratedKey(8073554114801607394928746621229606383, > 34393734663734382d316330302d346164372d613333372d3162 > 34303866613933333832) in /var/lib/cassandra/data/DFS/main-f-177-Data.db > java.io.EOFException > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267) > : > > nodetool netstats reports: > > Mode: Normal > Not sending any streams. > Not receiving any streams. > Pool Name Active Pending Completed > Commands n/a 0 526207 > Responses n/a 0 1747991
