The quickest solution is definitely going to be just blowing away the Hint files from the system keyspace data directories.
On Fri, Mar 26, 2010 at 5:36 PM, Scott White <scottbl...@gmail.com> wrote: > Nope it's always been random. > > On Fri, Mar 26, 2010 at 2:13 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Did you switch partitioner types at some point? >> >> On Fri, Mar 26, 2010 at 2:53 PM, Scott White <scottbl...@gmail.com> wrote: >> > I don't know if this is from switching from 0.5 to 0.6-betarc3 just >> > recently >> > or from doing a series of bootstrap and removeToken operations but I >> > recently started getting ArrayIndexOutOfBoundsException exceptions >> > (centered >> > around reading UTF from SSTableSliceIterator) on one of the machines in >> > my >> > cluster (see below). Obviously I could take the node out of the cluster >> > and >> > replace it with a new one, but if anyone has suggestions for how to >> > repair >> > the existing machine, that would be appreciated. >> > >> > thanks! >> > Scott >> > >> > Here are some examples: >> > >> > ERROR [ROW-READ-STAGE:3503] 2010-03-26 06:04:11,421 CassandraDaemon.java >> > (line 78) Fatal exception in thread Thread[ROW-READ-STAGE:3503,5,main] >> > java.lang.ArrayIndexOutOfBoundsException >> > at >> > >> > org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317) >> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381) >> > at java.io.DataInputStream.readUTF(DataInputStream.java:592) >> > at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887) >> > at >> > >> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125) >> > at >> > >> > org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59) >> > at >> > >> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722) >> > at org.apache.cassandra.db.Table.getRow(Table.java:381) >> > at >> > >> > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59) >> > at >> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80) >> > at >> > >> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:619) >> > >> > at >> > >> > org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317) >> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381) >> > at java.io.DataInputStream.readUTF(DataInputStream.java:592) >> > at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887) >> > at >> > >> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125) >> > at >> > >> > org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59) >> > at >> > >> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753) >> > at >> > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722) >> > at >> > >> > org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122) >> > at >> > >> > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250) >> > at >> > >> > org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80) >> > at >> > >> > org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280) >> > at >> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > >
