A little feedback, I scrubbed on each server and I haven't seen this error again. The load on each server eems to be correct. nodetool compactionstats shows boat-load of "Scrub" at 100% on my 3rd node but not on the 2 others. I left it that way and haven't restart yet.
2011/9/26 aaron morton <aa...@thelastpickle.com> > Looks like a mismatch between the key the index says should be at a certain > position in the date file and the key that is actually there. > > I've not checked but scrub *may* fix this this. Try it and see. > > (repair is for repairing consistency between nodes, scrub fixes local > issues with data. ) > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 26/09/2011, at 12:53 PM, Philippe wrote: > > Juste did > Could there be data corruption or will repairs do this? > > Thanks > Le 25 sept. 2011 15:30, "Jonathan Ellis" <jbel...@gmail.com> a écrit : > > Assertion errors are bugs, so that should worry you. > > > > However, I'd upgrade before filing a ticket. There were a lot of > > fixes in 0.8.5. > > > > On Sun, Sep 25, 2011 at 2:27 AM, Philippe <watche...@gmail.com> wrote: > >> Hello, > >> I've seen a couple of these in my logs, running 0.8.4. > >> This is a RF=3, 3-node cluster. 2 nodes including this one are on 0.8.4 > and > >> one is on 0.8.5 > >> > >> The node is still functionning hours later. Should I be worried ? > >> > >> Thanks > >> > >> ERROR [ReadStage:94911] 2011-09-24 22:40:30,043 > AbstractCassandraDaemon.java > >> (line 134) Fatal exception in thread Thread[ReadStage:94911,5,main] > >> java.lang.AssertionError: > >> > DecoratedKey(Token(bytes[0000224ceb80b5fb11e00000848783ceb9bf0000000000000002ffffffffffffff33]), > >> 0000224ceb80b5fb11e00000848783ceb9bf0000000000000002ffffffffffffff33) != > >> DecoratedKey(Token(bytes[038453154cb00000005f14]), > 038453154cb00000005f14) > >> in /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-10634-Data.db > >> at > >> > org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59) > >> at > >> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) > >> at > >> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169) > >> at org.apache.cassandra.db.Table.getRow(Table.java:385) > >> at > >> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58) > >> at > >> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) > >> at > >> > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:662) > >> ERROR [ReadStage:94936] 2011-09-24 22:40:30,042 > AbstractCassandraDaemon.java > >> (line 134) Fatal exception in thread Thread[ReadStage:94936,5,main] > >> java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) != > >> DecoratedKey(Token(bytes[038453154c900000005f14]), > 038453154c900000005f14) > >> in /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-10634-Data.db > >> at > >> > org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59) > >> at > >> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) > >> at > >> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1315) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1182) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169) > >> at org.apache.cassandra.db.Table.getRow(Table.java:385) > >> at > >> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58) > >> at > >> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) > >> at > >> > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1107) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:662) > >> ERROR [ReadStage:94713] 2011-09-24 22:40:30,041 > AbstractCassandraDaemon.java > >> (line 134) Fatal exception in thread Thread[ReadStage:94713,5,main] > >> java.lang.AssertionError: > >> > DecoratedKey(Token(bytes[7c4831fe0000000000000001ffffffffffffffaa000c6c697665626f782d6265306580000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffb0000c6c697665626f782d6631326380000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffab000b062a27f9e3000000005f13000000]), > >> > 7c4831fe0000000000000001ffffffffffffffaa000c6c697665626f782d6265306580000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffb0000c6c697665626f782d6631326380000000800000000000000000000002000700031195010000048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe00000000000000010000000000000001000700031195010001048000000000000000000001327e62362a00000024000100009c0dc550c60111e0000001687c4831fe0000000000000001ffffffffffffffab000b062a27f9e3000000005f13000000) > >> != DecoratedKey(Token(bytes[038451bfeeb00000005f14]), > >> 038451bfeeb00000005f14) in > >> /var/lib/cassandra/data/XXXXXXXXX/PUBLIC_MONTHLY_20-g-12305-Data.db > >> > >> > >> > >> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of DataStax, the source for professional Cassandra support > > http://www.datastax.com > > >
