I'm in the process of migrating data over to cassandra for several of our apps, and a few of the schemas use secondary indexes. Four times in the last couple months I've run into a corrupted sstable belonging to a secondary index, but have never seen this on any other sstables. When it happens, any query against the secondary index just hangs until the node is fixed. It's making me a bit nervous about using secondary indexes in production.
This has usually happened after a bulk data import, so I am wondering if the "firehose method" of dumping initial data into cassandra (write consistency = any) is causing some sort of write concurrency issue when it comes to secondary indexes. Has anyone else experienced this? The cluster is running 1.2.16 on 4x EC2 m1.large instances.
