> I have also been changing types, e.g. lock_tokens__ from MAP<UUID, BIGINT> > to MAP<TEXT, BIGINT>. The error looks like the schema was changed and a log replayed from before the change. Which obviously is not something we would expect to happen. Do you change the map type using ALTER TABLE (not sure if that is possible) or dropping / re-creating? > BTW, would a drain before running '/etc/init.d/cassandra stop' have helped? Probably not. Drain is designed to stop processing writes, flush everything from memory to disk and mark the commit log segments as no longer needed.
If you notice it again can you take note of the order of operations around dropping, creating or modifying the schema and restarting ? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 7/05/2013, at 5:03 AM, Blair Zajac <bl...@orcaware.com> wrote: > Hi Aaron, > > The keyspace consistent of 3 column families for user management, see below. > > I have dropped these tables multiple times since I'm testing a script to > automatically create the column families if they do not exists. I have also > been changing types, e.g. lock_tokens__ from MAP<UUID, BIGINT> to MAP<TEXT, > BIGINT>. > > I have tar copies of /var/lib/cassandra from all three nodes if somebody > wants to look. Since making the tarballs, I blew the cluster away and > re-initialized it from scratch. > > BTW, would a drain before running '/etc/init.d/cassandra stop' have helped? > > Regards, > Blair > > > CREATE TABLE account ( > pk_account UUID PRIMARY KEY, > last_login_using TEXT, > first_name TEXT, > last_name TEXT, > full_name TEXT, > created_micros BIGINT, > modified_micros BIGINT, > lock_tokens__ MAP<TEXT, BIGINT> > ); > > > CREATE TABLE external_account ( > pk_external_username TEXT PRIMARY KEY, > pk_account UUID, > primary_email_address TEXT, > secondary_email_addresses SET<TEXT>, > first_name TEXT, > last_name TEXT, > full_name TEXT, > last_login_micros BIGINT, > created_micros BIGINT, > modified_micros BIGINT, > lock_tokens__ MAP<TEXT, BIGINT> > ); > > CREATE TABLE email_address ( > pk_email_address TEXT PRIMARY KEY, > pk_account UUID, > pk_external_username SET<TEXT>, > lock_tokens__ MAP<TEXT, BIGINT> > ); > > > On 05/06/2013 01:14 AM, aaron morton wrote: >> Do you have the table definitions ? >> Any example data? >> Something is confused about a set / map / list type. >> >> It's failing when replying the log, if you want to work around move the >> commit log file out of the directory. There is a chance of data loss if >> this row mutation is being replied on all nodes. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 3/05/2013, at 2:36 PM, Blair Zajac <bl...@orcaware.com >> <mailto:bl...@orcaware.com>> wrote: >> >>> Hello, >>> >>> I'm running a 3-node development cluster on OpenStack VMs and recently >>> updated to DataStax's 1.2.4 debs on Ubuntu Raring after which the >>> cluster was fine. I shut it down for a few days and after getting >>> back to Cassandra today and booting the VMs, Cassandra is unable to >>> start. Below is the output from output.log from one of the nodes. >>> None of the Cassandra nodes can start. >>> >>> The deployment is pretty simple, two test keyspaces with a few column >>> families in each keyspace. I am doing a lot of keyspace and column >>> family deletions as I'm testing some db style migration code to >>> auto-setup a schema. >>> >>> Any suggestions? >>> >>> Blair >>> >>> INFO 19:24:09,780 Logging initialized >>> INFO 19:24:09,790 JVM vendor/version: Java HotSpot(TM) 64-Bit Server >>> VM/1.7.0_21 >>> INFO 19:24:09,791 Heap size: 880803840/880803840 >>> INFO 19:24:09,791 Classpath: >>> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/netty-3.5.9.Final.jar:/usr/share/cass! > and! >>> ra/lib/ser >>> vlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.4.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.4.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar >>> INFO 19:24:09,987 JNA mlockall successful >>> INFO 19:24:10,001 Loading settings from file:/etc/cassandra/cassandra.yaml >>> INFO 19:24:10,371 Data files directories: [/var/lib/cassandra/data] >>> INFO 19:24:10,372 Commit log directory: /var/lib/cassandra/commitlog >>> INFO 19:24:10,372 DiskAccessMode 'auto' determined to be mmap, >>> indexAccessMode is mmap >>> INFO 19:24:10,372 disk_failure_policy is stop >>> INFO 19:24:10,377 Global memtable threshold is enabled at 280MB >>> INFO 19:24:10,474 Not using multi-threaded compaction >>> INFO 19:24:10,816 Initializing key cache with capacity of 42 MBs. >>> INFO 19:24:10,822 Scheduling key cache save to each 14400 seconds >>> (going to save all keys). >>> INFO 19:24:10,823 Initializing row cache with capacity of 0 MBs and >>> provider org.apache.cassandra.cache.SerializingCacheProvider >>> INFO 19:24:10,827 Scheduling row cache save to each 0 seconds (going >>> to save all keys). >>> INFO 19:24:10,958 Opening >>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-165 >>> (35 bytes) >>> INFO 19:24:10,989 Opening >>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-166 >>> (168 bytes) >>> INFO 19:24:10,991 Opening >>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ib-164 >>> (346 bytes) >>> INFO 19:24:10,999 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db >>> INFO 19:24:11,018 Opening >>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-461 >>> (6562 bytes) >>> INFO 19:24:11,024 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db >>> INFO 19:24:11,031 Opening >>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-394 >>> (465 bytes) >>> INFO 19:24:11,032 Opening >>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-395 >>> (244 bytes) >>> INFO 19:24:11,033 Opening >>> /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ib-393 >>> (3025 bytes) >>> INFO 19:24:11,035 reading saved cache >>> /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db >>> INFO 19:24:11,048 Opening >>> /var/lib/cassandra/data/system/peers/system-peers-ib-16 (295 bytes) >>> INFO 19:24:11,052 Opening >>> /var/lib/cassandra/data/system/peers/system-peers-ib-15 (309 bytes) >>> INFO 19:24:11,057 Opening >>> /var/lib/cassandra/data/system/peers/system-peers-ib-14 (10840 bytes) >>> INFO 19:24:11,074 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-58 (97 bytes) >>> INFO 19:24:11,076 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-57 (97 bytes) >>> INFO 19:24:11,082 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-61 (150 bytes) >>> INFO 19:24:11,086 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-56 (148 bytes) >>> INFO 19:24:11,087 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-60 (97 bytes) >>> INFO 19:24:11,089 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-53 (5846 bytes) >>> INFO 19:24:11,091 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-55 (5410 bytes) >>> INFO 19:24:11,094 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-59 (97 bytes) >>> INFO 19:24:11,095 Opening >>> /var/lib/cassandra/data/system/local/system-local-ib-54 (120 bytes) >>> INFO 19:24:11,097 reading saved cache >>> /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db >>> INFO 19:24:11,666 Opening >>> /var/lib/cassandra/data/system_auth/users/system_auth-users-ib-1 (64 >>> bytes) >>> INFO 19:24:11,708 completed pre-loading (17 keys) key cache. >>> INFO 19:24:11,732 Replaying >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867617.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867618.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867619.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867620.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867621.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867622.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867623.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867624.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867625.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867626.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867627.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367547324162.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367547728748.log, >>> /var/lib/cassandra/commitlog/CommitLog-2-1367547828579.log >>> INFO 19:24:11,783 Replaying >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log >>> INFO 19:24:11,870 Finished reading >>> /var/lib/cassandra/commitlog/CommitLog-2-1367034867616.log >>> java.lang.RuntimeException: 7365636f6e646172795f656d61696c73 is not >>> defined as a collection >>> at >>> org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69) >>> at >>> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81) >>> at >>> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) >>> at java.util.TreeMap.put(TreeMap.java:545) >>> at >>> org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102) >>> at >>> org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88) >>> at >>> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109) >>> at >>> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104) >>> at >>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101) >>> at >>> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376) >>> at >>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203) >>> at >>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98) >>> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) >>> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:279) >>> at >>> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:356) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:601) >>> at >>> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212) >>> Cannot load daemon >> >
