Hi,
I'm getting an exception when doing many writes to a column with an
empty column name (i.e. empty byte buffer). The exception occurs during
compactation. It is reproducable given enough writes (around 1300000).
Is this a known issue? After the problem occurs cassandra will no longer
start up. It starts up again after I've manually deleted the commit log
files.
I have failed to reproduce the problem with a non empty byte array.
I'm using cassandra 0.7
Is there a way to recover from such errors without loosing the entries
in the commit log?
Regards
Mikael Wikblom
Here is my unit test, the exception and the yaml. I also provided the
stacktrace from the following startup exception
===unit test==============================
@Test
public void testInvalidColumnNames()
throws Exception
{
final Charset CHARSET = Charset.forName("UTF-8");
int count = 1300000;
for (int i = 0; i < count; i++)
{
List<RowMutation> mutations = new ArrayList<RowMutation>();
RowMutation m = new RowMutation("Test",
ByteBuffer.wrap(("rowId").getBytes(CHARSET)));
// QueryPath qp = new QueryPath("TestCassandra", null,
ByteBuffer.wrap(("test" + i).getBytes(CHARSET)));
QueryPath qp = new QueryPath("TestCassandra", null,
FBUtilities.EMPTY_BYTE_BUFFER);
m.add(qp, ByteBuffer.wrap("test".getBytes(CHARSET)),
System.currentTimeMillis());
mutations.add(m);
StorageProxy.mutate(mutations, ConsistencyLevel.ONE);
}
StorageService.instance.forceTableFlush(CassandraInitializer.KS,
"TestCassandra");
Thread.sleep(30*1000);
}
===exception when running test=========================================
13:54:38 ERROR [PrecompactedRow.<init>] Skipping row
DecoratedKey(rowId, 726f774964) in
./target/test/cassandra/data/Test/TestCassandra-e-4-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:137)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:132)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:312)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
===yaml file============================================
# Cassandra YAML generated from previous config
# Configuration wiki: http://wiki.apache.org/cassandra/StorageConfiguration
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
auto_bootstrap: false
binary_memtable_throughput_in_mb: 256
cluster_name: CluperUnitTest
column_index_size_in_kb: 64
commitlog_directory: ./target/test/cassandra/commitlog
commitlog_rotation_threshold_in_mb: 128
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
compaction_thread_priority: 1
concurrent_reads: 8
concurrent_writes: 32
data_file_directories:
- ./target/test/cassandra/data
disk_access_mode: auto
dynamic_snitch: false
dynamic_snitch_badness_threshold: 0.0
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_update_interval_in_ms: 100
endpoint_snitch: org.apache.cassandra.locator.RackInferringSnitch
hinted_handoff_enabled: true
in_memory_compaction_limit_in_mb: 256
index_interval: 128
saved_caches_directory: ./target/test/cassandra/saved_caches
keyspaces:
- column_families:
- column_metadata: []
column_type: Standard
compare_with: UTF8Type
gc_grace_seconds: 864000
key_cache_save_period_in_seconds: 3600
keys_cached: 200000.0
max_compaction_threshold: 32
memtable_flush_after_mins: 60
memtable_operations_in_millions: 0.3
memtable_throughput_in_mb: 64
min_compaction_threshold: 4
name: TestCassandra
read_repair_chance: 1.0
row_cache_save_period_in_seconds: 0
rows_cached: 0.0
name: Test
replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
replication_factor: 1
listen_address: localhost
partitioner: org.apache.cassandra.dht.OrderPreservingPartitioner
phi_convict_threshold: 8
rpc_keepalive: true
rpc_port: 9161
rpc_timeout_in_ms: 10000
seeds:
- 127.0.0.1
sliced_buffer_size_in_kb: 64
snapshot_before_compaction: false
storage_port: 7012
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
===start up exception after unit test===================================
14:16:14 ERROR [AbstractCassandraDaemon.activate] Exception encountered
during startup.
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
at
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
at
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
at
senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
at java.lang.Thread.run(Thread.java:662)
at
senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
Exception encountered during startup.
at java.lang.Thread.run(Thread.java:662)'
===ls of the commit log directory after the error=================
mikael@mikael2:~/projects/sitevision/trunk/framework/sitevision-framework-core/target/test-classes/target/test/cassandra/commitlog>
ls -ltr
total 187936
-rw-r--r-- 1 mikael wheel 134217730 2011-02-01 14:13
CommitLog-1296565996491.log
-rw-r--r-- 1 mikael wheel 28 2011-02-01 14:13
CommitLog-1296566017932.log.header
-rw-r--r-- 1 mikael wheel 52 2011-02-01 14:13
CommitLog-1296565996491.log.header
-rw-r--r-- 1 mikael wheel 58211952 2011-02-01 14:13
CommitLog-1296566017932.log
-rw-r--r-- 1 mikael wheel 0 2011-02-01 14:16
CommitLog-1296566174499.log
-rw-r--r-- 1 mikael wheel 20 2011-02-01 14:16
CommitLog-1296566174499.log.header
--
Mikael Wikblom
Software Architect
SiteVision AB
019-217058
mikael.wikb...@sitevision.se
http://www.sitevision.se
