First column in the row has offset in the file of 190226525, last valid column is at 380293592, about 181MB from first column to last.
in_memory_compaction_limit was 128MB, so almost certainly above the limit. Terje On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen <tmarthinus...@gmail.com > wrote: > In my case, probably yes. From thw rows I have looked at, I think I have > only seen this on rows with 1 million plus columns/supercolumns. > > May very well been larger than in memory limit. I think the compacted row I > looked closer at was about 200MB and the in memory limit may have been > 256MB. > > I will see if we still got files around to verify. > > Regards, > Terje > > On 26 Apr 2011, at 02:08, Jonathan Ellis <jbel...@gmail.com> wrote: > > > Was it on a "large" row? (> in_memory_compaction_limit?) > > > > I'm starting to suspect that LazilyCompactedRow is computing row size > > incorrectly in some cases. > > > > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen > > <tmarthinus...@gmail.com> wrote: > >> I have been hunting similar looking corruptions, especially in the hints > >> column family, but I believe it occurs somewhere while compacting. > >> I looked in greater detail on one sstable and the row length was longer > than > >> the actual data in the row, and as far as I could see, either the length > was > >> wrong or the row was missing data as there was was no extra data in the > row > >> after the last column. > >> This was however on a somewhat aging dataset, so suspected it could be > >> related to 2376. > >> > >> Playing around with 0.8 at the moment and not seen it there yet.... (bet > it > >> will show up tomorrow once I wrote that.. :)) > >> Terje > >> > >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni < > sanj...@locomatix.com> > >> wrote: > >>> > >>> Hi Sylvain, > >>> I started it from 0.7.4 with the patch 2376. No upgrade. > >>> Thanks! > >>> > >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne < > sylv...@datastax.com> > >>> wrote: > >>>> > >>>> Hi Sanjeev, > >>>> > >>>> What's the story of the cluster ? Did you started with 0.7.4, or is it > >>>> upgraded from > >>>> some earlier version ? > >>>> > >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni < > sanj...@locomatix.com> > >>>> wrote: > >>>>> Hey guys, > >>>>> Running a one node cassandra server with version 0.7.4 patched > >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376 > >>>>> The system was running fine for a couple of days when we started > >>>>> noticing > >>>>> something strange with cassandra. I stopped all applications and > >>>>> restarted > >>>>> cassandra. And then did a scrub. During scrub, I noticed these in the > >>>>> logs > >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561 > >>>>> CompactionManager.java > >>>>> (line 607) Non-fatal error reading row (stacktrace follows) > >>>>> java.io.IOError: java.io.IOException: Impossible row size > >>>>> 1516029079813320210 > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) > >>>>> at > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >>>>> at > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>> at java.lang.Thread.run(Thread.java:662) > >>>>> Caused by: java.io.IOException: Impossible row size > 1516029079813320210 > >>>>> ... 8 more > >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640 > >>>>> CompactionManager.java > >>>>> (line 613) Retrying from row index; data is -1768177699 bytes > starting > >>>>> at > >>>>> 2626524914 > >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641 > >>>>> CompactionManager.java > >>>>> (line 633) Retry failed too. Skipping to next row (retry's > stacktrace > >>>>> follows) > >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be > >>>>> 1868982636 > >>>>> bytes, longer than entire row size -1768177699 at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) > >>>>> at > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>> at java.lang.Thread.run(Thread.java:662) > >>>>> Caused by: java.io.EOFException: bloom filter claims to be 1868982636 > >>>>> bytes, > >>>>> longer than entire row size -1768177699 at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87) > >>>>> ... 8 more > >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545 > >>>>> CompactionManager.java > >>>>> (line 607) Non-fatal error reading row (stacktrace follows) > >>>>> java.io.IOError: java.io.EOFException > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > >>>>> at > >>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76) > >>>>> at > >>>>> > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) > >>>>> at > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>> at java.lang.Thread.run(Thread.java:662) > >>>>> Caused by: java.io.EOFException > >>>>> at > >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) > >>>>> at > >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270) > >>>>> at > >>>>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140) > >>>>> ... 22 more > >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561 > >>>>> CompactionManager.java > >>>>> (line 613) Retrying from row index; data is 78540539 bytes starting > at > >>>>> 2229643127 > >>>>> > >>>>> And then when i restarted the readers, i get the following crash > >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658 > >>>>> AbstractCassandraDaemon.java > >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main] > >>>>> java.lang.AssertionError: mmap segment underflow; remaining is > >>>>> 791462117 but > >>>>> 1970433058 requested > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119) > >>>>> at > >>>>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) > >>>>> at > >>>>> > >>>>> > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > >>>>> at > >>>>> > >>>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173) > >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63) > >>>>> at > >>>>> > >>>>> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453) > >>>>> at > >>>>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>>> at > >>>>> > >>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>>> at java.lang.Thread.run(Thread.java:662) > >>>>> > >>>>> Any ideas? > >>>>> Thanks! > >>> > >> > >> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of DataStax, the source for professional Cassandra support > > http://www.datastax.com >
