Do you have the table definitions ? Any example data? Something is confused about a set / map / list type.
It's failing when replying the log, if you want to work around move the commit log file out of the directory. There is a chance of data loss if this row mutation is being replied on all nodes. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 3/05/2013, at 2:36 PM, Blair Zajac <bl...@orcaware.com> wrote: > Hello, > > I'm running a 3-node development cluster on OpenStack VMs and recently > updated to DataStax's 1.2.4 debs on Ubuntu Raring after which the cluster was > fine. I shut it down for a few days and after getting back to Cassandra > today and booting the VMs, Cassandra is unable to start. Below is the output > from output.log from one of the nodes. None of the Cassandra nodes can start. > > The deployment is pretty simple, two test keyspaces with a few column > families in each keyspace. I am doing a lot of keyspace and column family > deletions as I'm testing some db style migration code to auto-setup a schema. > > Any suggestions? > > Blair > > INFO 19:24:09,780 Logging initialized > INFO 19:24:09,790 JVM vendor/version: Java HotSpot(TM) 64-Bit Server > VM/1.7.0_21 > INFO 19:24:09,791 Heap size: 880803840/880803840 > INFO 19:24:09,791 Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/netty-3.5.9.Final.jar:/usr/share/cassand! > ra/lib/ser > vlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.4.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.4.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 19:24:09,987 JNA mlockall successful > INFO 19:24:10,001 Loading settings from file:/etc/cassandra/cassandra.yaml > INFO 19:24:10,371 Data files directories: [/var/lib/cassandra/data] > INFO 19:24:10,372 Commit log directory: /var/lib/cassandra/commitlog > INFO 19:24:10,372 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 19:24:10,372 disk_failure_policy is stop > INFO 19:24:10,377 Global memtable threshold is enabled at 280MB > INFO 19:24:10,474 Not using multi-threaded compaction > INFO 19:24:10,816 Initializing key cache with capacity of 42 MBs. > INFO 19:24:10,822 Scheduling key cache save to each 14400 seconds (going to > save all keys). > INFO 19:24:10,823 Initializing row cache with capacity of 0 MBs and provider > org.apache.cassandra.cache.SerializingCacheProvider > INFO 19:24:10,827 Scheduling row cache save to each 0 seconds (going to save > all keys). > INFO 19:24:10,958 Opening > /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-165 > (35 bytes) > INFO 19:24:10,989 Opening > /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-166 > (168 bytes) > INFO 19:24:10,991 Opening > /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-164 > (346 bytes) > INFO 19:24:10,999 reading saved cache > /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db > INFO 19:24:11,018 Opening > /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-461 > (6562 bytes) > INFO 19:24:11,024 reading saved cache > /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db > INFO 19:24:11,031 Opening > /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-394 > (465 bytes) > INFO 19:24:11,032 Opening > /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-395 > (244 bytes) > INFO 19:24:11,033 Opening > /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-393 > (3025 bytes) > INFO 19:24:11,035 reading saved cache > /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db > INFO 19:24:11,048 Opening > /var/lib/cassandra/data/system/peers/system-peers-ib-16 (295 bytes) > INFO 19:24:11,052 Opening > /var/lib/cassandra/data/system/peers/system-peers-ib-15 (309 bytes) > INFO 19:24:11,057 Opening > /var/lib/cassandra/data/system/peers/system-peers-ib-14 (10840 bytes) > INFO 19:24:11,074 Opening > /var/lib/cassandra/data/system/local/system-local-ib-58 (97 bytes) > INFO 19:24:11,076 Opening > /var/lib/cassandra/data/system/local/system-local-ib-57 (97 bytes) > INFO 19:24:11,082 Opening > /var/lib/cassandra/data/system/local/system-local-ib-61 (150 bytes) > INFO 19:24:11,086 Opening > /var/lib/cassandra/data/system/local/system-local-ib-56 (148 bytes) > INFO 19:24:11,087 Opening > /var/lib/cassandra/data/system/local/system-local-ib-60 (97 bytes) > INFO 19:24:11,089 Opening > /var/lib/cassandra/data/system/local/system-local-ib-53 (5846 bytes) > INFO 19:24:11,091 Opening > /var/lib/cassandra/data/system/local/system-local-ib-55 (5410 bytes) > INFO 19:24:11,094 Opening > /var/lib/cassandra/data/system/local/system-local-ib-59 (97 bytes) > INFO 19:24:11,095 Opening > /var/lib/cassandra/data/system/local/system-local-ib-54 (120 bytes) > INFO 19:24:11,097 reading saved cache > /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db > INFO 19:24:11,666 Opening > /var/lib/cassandra/data/system_auth/users/system_auth-users-ib-1 (64 bytes) > INFO 19:24:11,708 completed pre-loading (17 keys) key cache. > INFO 19:24:11,732 Replaying > /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867617.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867618.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867619.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867620.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867621.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867622.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867623.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867624.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867625.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867626.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367034867627.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367547324162.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367547728748.log, > /var/lib/cassandra/commitlog/CommitLog-2-1367547828579.log > INFO 19:24:11,783 Replaying > /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log > INFO 19:24:11,870 Finished reading > /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log > java.lang.RuntimeException: 7365636f6e646172795f656d61696c73 is not defined > as a collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) > at java.util.TreeMap.put(TreeMap.java:545) > at > org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102) > at > org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88) > at > org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109) > at > org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101) > at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:279) > at > org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:356) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212) > Cannot load daemon
