> We deleted and recreated those CFs before moving into > production mode. We have a wiener.
The comparator is applying the current schema to the byte value read from disk (schema on read) which describes a value with more than 2 components. It's then trying to apply the current schema so it can type cast the bytes for comparison. Something must have gone wrong in the "deleted" part of your statement above. We do not store schema with data, so this a problem of changing the schema in an incompatible way with existing data. nodetool scrub is probably your best bet. I've not checked that it handles this specific problem, but in general it will drop rows from SSTables that cannot be read or have some other problem. Best thing to do is snapshot and copy the data from one prod node to a QA box and run some tests. hope that helps. ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 3/04/2013, at 2:11 AM, Dane Miller <d...@optimalsocial.com> wrote: > On Mon, Apr 1, 2013 at 10:19 PM, aaron morton <aa...@thelastpickle.com> wrote: >> ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line >> 133) Exception in thread Thread[Thread-232,5,main] >> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2) >> at >> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) >> at >> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) >> at >> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81) >> at >> org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96) >> >> Something odd in the schema world perhaps. >> >> Has the schema changed recently? > > No, not recently. But during development we experimented with other > comparator types for those CFs. More info below. > >> Do yo have more than one schema in the cluster ? (describe cluster in >> cassandra-cli) > > I don't think so, that command in cassandra-cli shows just a single > schema in the cluster: > > Cluster Information: > Snitch: org.apache.cassandra.locator.Ec2Snitch > Partitioner: org.apache.cassandra.dht.RandomPartitioner > Schema versions: > 126b31ad-3660-3831-9d4f-c6763c9acc97: [ ...ip list... ] > > > This error happened on a CF where the composite is: (Integer, UTF8) > > I'm a bit stumped about how we could get the an index==2 in that code > pathway. See here: > https://github.com/apache/cassandra/blob/cassandra-1.2.1/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java > > ...start at line 63 in compare. > > My Java is terrible, but all of our CompositeTypes are composites of > only two types. Thus the counter 'i' should never get up to 2 (it is > used to access by index to individual comparator types within the > composite), unless the value of the first two components in each of > the column names being compared are equal, which should be impossible. > > During development we experimented with other comparator types for > those CFs. We deleted and recreated those CFs before moving into > production mode. Is there a chance Cassandra 'remembers' these old > types from a deleted CF that shares a name with an existing CF? Could > that be causing improper parsing of comparator column names? > > That we enter this code pathway from a section that seems to want to > clean up tombstones makes me think this is a possibility, that there > is a tombstone somewhere whose composite column name is causing > issues. > > Dane
