Thanks for the feedback. This node actually shut down half way when it was bootstrapping the first time which likely led to this data corruption. We restarted the JVM and it appeared stable until this issue. We decided to stop cassandra, wipe the node, and restart so that it can bootstrap again to ensure all data is "clean".
From: Ben Coverston <ben.covers...@datastax.com<mailto:ben.covers...@datastax.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Monday, August 5, 2013 11:23 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: org.apache.cassandra.io.sstable.CorruptSSTableException Also check your system log for IO Errors. Scrub may eliminate the error, but even if it does work you should still run repair. This type of corruption usually happens because of a failed or failing disk/memory. On Mon, Aug 5, 2013 at 8:44 AM, Jason Wee <peich...@gmail.com<mailto:peich...@gmail.com>> wrote: you can try nodetool scrub. if it does not work, try repair then cleanup. had this issue a few weeks back but our version is 1.0.x On Mon, Aug 5, 2013 at 8:12 AM, Keith Wright <kwri...@nanigans.com<mailto:kwri...@nanigans.com>> wrote: Re-sending hoping to get some help. Any ideas would be much appreciated! From: Keith Wright <kwri...@nanigans.com<mailto:kwri...@nanigans.com>> Date: Friday, August 2, 2013 3:01 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: org.apache.cassandra.io.sstable.CorruptSSTableException Hi all, We just added a node to our cluster (1.2.4 Vnodes) and they appear to be running well exception I see that the new node is not making any progress compacting one of the CF. The exception below is generated. My assumption is that the only way to handle this is to stop the node, delete the file in question, restart, and run repair. Thoughts? org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 1249463589142530 starting at 5604968 would be larger than file /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db length 14017479 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:168) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:83) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:152) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:139) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:36) at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:288) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: dataSize of 1249463589142530 starting at 5604968 would be larger than file /data/3/cassandra/data/users/global_user/users-global_user-ib-1550-Data.db length 14017479 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:123) ... 9 more -- Ben Coverston DataStax -- The Apache Cassandra Company
