[
https://issues.apache.org/jira/browse/CASSANDRA-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165249#comment-14165249
]
Aleksey Yeschenko commented on CASSANDRA-7188:
----------------------------------------------
bq. Maybe it is related to the fact that CQL doesn't support well super column
families?
Nope, CQL supports super columns very well. The bug here is in the
composites<->supercolumns conversion, caused by CASSANDRA-3237.
A very trivial one, too - to serialize a CounterMutation for forwarding it to a
1.2 node, if that one is chosen a leader by the 2.0 coordinator, as part of the
process we call Column#withUpdatedName() - in SuperColumns#groupSuperColumns(),
in this case. And CounterUpdateColumn was missing a withUpdatedName() override,
so when serializing, it used the implementation in Column, and didn't set the
correct flag. The 1.2 node, upon receiving the serialized mutation, read those
columns as regular Column, and that's what's been causing your reconcile assert.
> Wrong class type: class org.apache.cassandra.db.Column in
> CounterColumn.reconcile
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-7188
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7188
> Project: Cassandra
> Issue Type: Bug
> Reporter: Nicolas Lalevée
> Assignee: Aleksey Yeschenko
> Attachments: 7188.txt
>
>
> When migrating a cluster of 6 nodes from 1.2.11 to 2.0.7, we started to see
> on the first migrated node this error:
> {noformat}
> ERROR [ReplicateOnWriteStage:1] 2014-05-07 11:26:59,779 CassandraDaemon.java
> (line 198) Exception in thread Thread[ReplicateOnWriteStage:1,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1540)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1369)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
> at
> org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1085)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1916)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We then saw on the other 5 nodes, still on 1.2.x, this error:
> {noformat}
> ERROR [MutationStage:2793] 2014-05-07 11:46:12,301 CassandraDaemon.java (line
> 191) Exception in thread Thread[MutationStage:2793,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
> at
> org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
> at
> org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
> at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
> at
> org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
> at
> org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
> at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:742)
> at org.apache.cassandra.db.Table.apply(Table.java:388)
> at org.apache.cassandra.db.Table.apply(Table.java:353)
> at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:280)
> at
> org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:137)
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:773)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1651)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> {noformat}
> Here some other stack we also on the 5 unmigrated nodes:
> {noformat}
> ERROR [ReadStage:4242] 2014-05-07 11:46:12,259 CassandraDaemon.java (line
> 191) Exception in thread Thread[ReadStage:4242,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
> at
> org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
> at
> org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
> at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> {noformat}
> And the client side, it is failing with:
> {noformat}
> Caused by: org.apache.cassandra.thrift.UnavailableException: null
> at
> org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7866)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:594)
> at
> org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:578)
> at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:274)
> {noformat}
> After seeing such errors, we just shut down the first migrated node, hoping
> it would avoid all these client errors. But errors continue to be logged,
> even if there were only the 5 1.2.x nodes in the ring.
> As the usual wild guess, let's reboot a node to fix it. At our damned
> surprise, it would restart and would fail with:
> {noformat}
> INFO 11:33:40,190 Initializing system.LocationInfo
> java.lang.AssertionError
> at
> org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162)
> at
> org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541)
> at
> org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456)
> at
> org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306)
> at
> org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287)
> at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154)
> at
> org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:574)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> Cannot load daemon
> Service exit with a return value of 3
> {noformat}
> From there we only had 4 running nodes, with errors spreading around. So we
> halted everything, put the first node back to 1.2.11 and restored the data
> which has been snapshot just before the first node was migrated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
