No, brand new, started from scratch, no data at all. My setup get into trouble after a couple of hours since it then starts giving the Dead/up messages. I also get messages like '11 MUTATION messages dropped in server lifetime' Later on I got this type of exception also.
-----Original Message----- From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: dinsdag 27 september 2011 17:41 To: user@cassandra.apache.org Subject: Re: invalid column name length 0 Is this upgraded from an earlier version? On Tue, Sep 27, 2011 at 8:51 AM, Desimpel, Ignace <ignace.desim...@nuance.com> wrote: > Version 0.8.6. > > After an extreme load to 4 (embedded) cassandra servers with > replication factor 3 ( Ubuntu 10.4, dual six core, 64 bit, no swap, 1 > 15000 rpm commitlog disk, 1 15000 rpm datafile disk, ) I get a fatal > exception as listed below. > > No more messages were found after that. > > > > Probably other things are going wrong like the message "410 Could not > complete hinted handoff to /xxx.yyy.zzz.60", or the Dead/Up messages > (occurring long before this exception). > > But maybe this exception can point me in the right direction or even > point out some bug in Cassandra. > > > > Thanks, > > Ignace > > > > > > 2011-09-27 06:59:46,383 Compacting large row > KsFullIdx/ForwardStringValues:3237343034 (178032211 bytes) > incrementally > > 2011-09-27 07:00:09,738 GC for ParNew: 311 ms for 1 collections, > 8139389704 used; max is 33344716800 > > 2011-09-27 07:00:12,818 Compacting large row > KsFullIdx/ForwardStringValues:31363437 (1281862723 bytes) > incrementally > > 2011-09-27 07:02:16,025 Compacting large row > KsFullIdx/ForwardStringValues:31363438 (1623095072 bytes) > incrementally > > 2011-09-27 07:04:38,332 GC for ParNew: 534 ms for 1 collections, > 7811259472 used; max is 33344716800 > > 2011-09-27 07:04:52,803 Compacting large row > KsFullIdx/ForwardStringValues:3238313433 (1435774436 bytes) > incrementally > > 2011-09-27 07:06:57,160 Compacted to > /media/datadrive1/capd.cassandra.capd/dbdatafile/KsFullIdx/ForwardStringValues-tmp-g-542-Data.db. > 43,244,902,670 to 42,780,624,408 (~98% of original) bytes for 1,260 keys. > Time: 4,321,960ms. > > 2011-09-27 08:01:42,090 Saved KsFullIdx-ForwardStringValues-KeyCache > (572 > items) in 16 ms > > 2011-09-27 08:01:42,182 Saved KsFullIdx-ReverseStringValues-KeyCache > (25688 > items) in 63 ms > > 2011-09-27 08:18:13,078 InetAddress /xxx.yyy.zzz.62 is now dead. > > 2011-09-27 08:18:16,467 InetAddress /xxx.yyy.zzz.62 is now UP > > 2011-09-27 08:48:56,410 Could not complete hinted handoff to > /xxx.yyy.zzz.60 > > 2011-09-27 08:48:56,410 Enqueuing flush of > Memtable-HintsColumnFamily@2083796703(12097/196566 serialized/live > bytes, > 254 ops) > > 2011-09-27 08:48:56,411 Writing > Memtable-HintsColumnFamily@2083796703(12097/196566 serialized/live > bytes, > 254 ops) > > 2011-09-27 08:48:56,411 Nothing to compact in HintsColumnFamily; use > forceUserDefinedCompaction if you wish to force compaction of single > sstables (e.g. for tombstone collection) > > 2011-09-27 08:48:56,411 Finished hinted handoff of 254 rows to > endpoint > /xxx.yyy.zzz.60 > > 2011-09-27 08:48:56,490 Completed flushing > /media/datadrive1/capd.cassandra.capd/dbdatafile/system/HintsColumnFam > ily-g-10-Data.db > (25079 bytes) > > 2011-09-27 08:49:42,858 Started hinted handoff for endpoint > /xxx.yyy.zzz.62 > > 2011-09-27 12:01:42,100 Saved KsFullIdx-ForwardStringValues-KeyCache > (712 > items) in 27 ms > > 2011-09-27 12:01:42,182 Saved KsFullIdx-ReverseStringValues-KeyCache > (30742 > items) in 55 ms > > 2011-09-27 12:10:01,016 InetAddress /xxx.yyy.zzz.59 is now dead. > > 2011-09-27 12:10:02,272 InetAddress /xxx.yyy.zzz.59 is now UP > > 2011-09-27 12:17:34,596 Fatal exception in thread > Thread[HintedHandoff:1,5,RMI Runtime] > > java.io.IOError: > org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: > invalid column name length 0 > > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSort > edMap.java:265) > > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java: > 281) > > at > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java: > 236) > > at > java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentS > kipListMap.java:1493) > > at > java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMa > p.java:1443) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn. > java:445) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn. > java:428) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn. > java:418) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn. > java:380) > > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlock > Fetcher.getNextBlock(IndexedSliceReader.java:179) > > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext( > IndexedSliceReader.java:121) > > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext( > IndexedSliceReader.java:49) > > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIt > erator.java:140) > > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.ja > va:135) > > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SS > TableSliceIterator.java:108) > > at > org.apache.commons.collections.iterators.CollatingIterator.set(Collati > ngIterator.java:283) > > at > org.apache.commons.collections.iterators.CollatingIterator.least(Colla > tingIterator.java:326) > > at > org.apache.commons.collections.iterators.CollatingIterator.next(Collat > ingIterator.java:230) > > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterat > or.java:69) > > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIt > erator.java:140) > > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.ja > va:135) > > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns( > SliceQueryFilter.java:116) > > at > org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(Quer > yFilter.java:142) > > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFam > ilyStore.java:1427) > > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily > Store.java:1304) > > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamily > Store.java:1261) > > at > org.apache.cassandra.db.HintedHandOffManager.sendRow(HintedHandOffMana > ger.java:155) > > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hi > ntedHandOffManager.java:350) > > at > org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffM > anager.java:89) > > at > org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandO > ffManager.java:397) > > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30 > ) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu > tor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: > invalid column name length 0 > > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer. > java:89) > > at > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSort > edMap.java:261) > > ... 33 more > > 2011-09-27 12:17:53,291 Started hinted handoff for endpoint > /xxx.yyy.zzz.59 -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
