Hello,
I'm running a 3-node development cluster on OpenStack VMs and recently
updated to DataStax's 1.2.4 debs on Ubuntu Raring after which the
cluster was fine. I shut it down for a few days and after getting back
to Cassandra today and booting the VMs, Cassandra is unable to start.
Below is the output from output.log from one of the nodes. None of the
Cassandra nodes can start.
The deployment is pretty simple, two test keyspaces with a few column
families in each keyspace. I am doing a lot of keyspace and column
family deletions as I'm testing some db style migration code to
auto-setup a schema.
Any suggestions?
Blair
INFO 19:24:09,780 Logging initialized
INFO 19:24:09,790 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
VM/1.7.0_21
INFO 19:24:09,791 Heap size: 880803840/880803840
INFO 19:24:09,791 Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/netty-3.5.9.Final.jar:/usr/share/cassand!
ra/lib/ser
vlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.4.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.4.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 19:24:09,987 JNA mlockall successful
INFO 19:24:10,001 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO 19:24:10,371 Data files directories: [/var/lib/cassandra/data]
INFO 19:24:10,372 Commit log directory: /var/lib/cassandra/commitlog
INFO 19:24:10,372 DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
INFO 19:24:10,372 disk_failure_policy is stop
INFO 19:24:10,377 Global memtable threshold is enabled at 280MB
INFO 19:24:10,474 Not using multi-threaded compaction
INFO 19:24:10,816 Initializing key cache with capacity of 42 MBs.
INFO 19:24:10,822 Scheduling key cache save to each 14400 seconds
(going to save all keys).
INFO 19:24:10,823 Initializing row cache with capacity of 0 MBs and
provider org.apache.cassandra.cache.SerializingCacheProvider
INFO 19:24:10,827 Scheduling row cache save to each 0 seconds (going
to save all keys).
INFO 19:24:10,958 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-165
(35 bytes)
INFO 19:24:10,989 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-166
(168 bytes)
INFO 19:24:10,991 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-164
(346 bytes)
INFO 19:24:10,999 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
INFO 19:24:11,018 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-461
(6562 bytes)
INFO 19:24:11,024 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
INFO 19:24:11,031 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-394
(465 bytes)
INFO 19:24:11,032 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-395
(244 bytes)
INFO 19:24:11,033 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-393
(3025 bytes)
INFO 19:24:11,035 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
INFO 19:24:11,048 Opening
/var/lib/cassandra/data/system/peers/system-peers-ib-16 (295 bytes)
INFO 19:24:11,052 Opening
/var/lib/cassandra/data/system/peers/system-peers-ib-15 (309 bytes)
INFO 19:24:11,057 Opening
/var/lib/cassandra/data/system/peers/system-peers-ib-14 (10840 bytes)
INFO 19:24:11,074 Opening
/var/lib/cassandra/data/system/local/system-local-ib-58 (97 bytes)
INFO 19:24:11,076 Opening
/var/lib/cassandra/data/system/local/system-local-ib-57 (97 bytes)
INFO 19:24:11,082 Opening
/var/lib/cassandra/data/system/local/system-local-ib-61 (150 bytes)
INFO 19:24:11,086 Opening
/var/lib/cassandra/data/system/local/system-local-ib-56 (148 bytes)
INFO 19:24:11,087 Opening
/var/lib/cassandra/data/system/local/system-local-ib-60 (97 bytes)
INFO 19:24:11,089 Opening
/var/lib/cassandra/data/system/local/system-local-ib-53 (5846 bytes)
INFO 19:24:11,091 Opening
/var/lib/cassandra/data/system/local/system-local-ib-55 (5410 bytes)
INFO 19:24:11,094 Opening
/var/lib/cassandra/data/system/local/system-local-ib-59 (97 bytes)
INFO 19:24:11,095 Opening
/var/lib/cassandra/data/system/local/system-local-ib-54 (120 bytes)
INFO 19:24:11,097 reading saved cache
/var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
INFO 19:24:11,666 Opening
/var/lib/cassandra/data/system_auth/users/system_auth-users-ib-1 (64 bytes)
INFO 19:24:11,708 completed pre-loading (17 keys) key cache.
INFO 19:24:11,732 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867617.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867618.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867619.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867620.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867621.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867622.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867623.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867624.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867625.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867626.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367034867627.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367547324162.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367547728748.log,
/var/lib/cassandra/commitlog/CommitLog-2-1367547828579.log
INFO 19:24:11,783 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
INFO 19:24:11,870 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log
java.lang.RuntimeException: 7365636f6e646172795f656d61696c73 is not
defined as a collection
at
org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at java.util.TreeMap.put(TreeMap.java:545)
at
org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
at
org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:279)
at
org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
Cannot load daemon
