Hi,
I'm taking a look at the Check and Set functionalities provided by the
cas() API provided by cassandra 2.0 (the code available on git). I'm
running a few tests on a small sized cluster (replication factor 3,
consistency level quorum) with a few clients. I've observed a lot of cases
seem to hit the TimedOutException while paxos is in progress. On inspection
of the server side logs the following stack was seen:
ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line 196)
Exception in thread Thread[Thread-582,5,main]
java.lang.AssertionError: Added column does not sort as the last column
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
at
org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
at
org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
at
org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
I was updating multiple columns using the same cas() call, and looking at
the assert (code below) I added code to sort them before sending.
However, now I'm seeing the same issue *even with single column* cas()
calls.
int c = internalComparator().compare(columns.get(getColumnCount() -
1).name(), column.name());
// note that we want an assertion here (see addColumn javadoc), but
we also want that if
// assertion are disabled, addColumn works correctly with unsorted
input
assert c <= 0 : "Added column does not sort as the " + (reversed ?
"first" : "last") + " column";
I have also seen this error occur in the getRow() path, as below:
java.lang.AssertionError: Added column does not sort as the last column
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
at
org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
at org.apache.cassandra.db.Table.getRow(Table.java:331)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Could someone provide me a little more clarity about what the context of
the error is and what the client needs to do to fix it? I understand that
running with assertions disabled would do away with this error, but i'd
like to know if there is a proper way to fix this. Moreover, what could
possibly be going wrong in the get path that is hitting this issue?
Some guidance, on the fact that whether it is expected that the client sort
the columns before calling the API with multiple columns, would be really
appreciated.
Thanks,
Soumava
