That would be a bug (as any assertion error would be), likely some race condition. Could you open a ticket? The fact that this block the MemtablePostFlusher is unfortunately related. Restarting the node would fix but we need to make that more solid too.
-- Sylvain On Thu, Nov 10, 2011 at 9:04 PM, Dan Hendry <dan.hendry.j...@gmail.com> wrote: > Just happened again, seems to be with the same column family (at least on a > flusher thread for which the last activity was flushing a memtable for that > CF). > > > > It also looks like MemtablePostFlusher tasks blocked (and not getting > cleared) as evidenced by tpstats: > > > > Pool Name Active Pending Completed Blocked All > time blocked > > MemtablePostFlusher 1 18 16 > 0 0 > > > > Dan > > > > From: Dan Hendry [mailto:dan.hendry.j...@gmail.com] > Sent: November-10-11 14:49 > To: 'user@cassandra.apache.org' > Subject: 1.0.2 Assertion Error > > > > Just saw this weird assertion after upgrading one of my nodes from 0.8.6 to > 1.0.2 (its been running fine for a few hours now): > > > > INFO [FlushWriter:9] 2011-11-10 13:08:58,882 Memtable.java (line 237) > Writing Memtable-Data@1388955390(25676955/430716097 serialized/live bytes, > 478913 ops) > > ERROR [FlushWriter:9] 2011-11-10 13:08:59,513 AbstractCassandraDaemon.java > (line 133) Fatal exception in thread Thread[FlushWriter:9,5,main] > > java.lang.AssertionError: CF size changed during serialization: was 4 > initially but 3 written > > at > org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:94) > > at > org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:112) > > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:177) > > at > org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:264) > > at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47) > > at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289) > > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > > > After taking a quick peek at the code, it looks like the numbers (“4 > initially but 3 written”) refer to the number of columns (in the memtable?). > Given the byte size of the memtable being flushed, there are *certainly* > more than 4 columns. Besides this error, there are no other unexpected or > unusual log entries and the node seems to be behaving normally. > > > > Thoughts? > > > > Dan Hendry
