I think the problem here is your test.
The CassandraServer class calls validateColumns() and will not let zero length
columns be inserted. The call would not get to the storage proxy.
Aaron
On 2/02/2011, at 2:27 AM, Mikael Wikblom <mikael.wikb...@sitevision.se> wrote:
> Hi,
>
> I'm getting an exception when doing many writes to a column with an
> empty column name (i.e. empty byte buffer). The exception occurs during
> compactation. It is reproducable given enough writes (around 1300000).
> Is this a known issue? After the problem occurs cassandra will no longer
> start up. It starts up again after I've manually deleted the commit log
> files.
>
> I have failed to reproduce the problem with a non empty byte array.
>
> I'm using cassandra 0.7
>
> Is there a way to recover from such errors without loosing the entries
> in the commit log?
>
> Regards
>
> Mikael Wikblom
>
>
>
> Here is my unit test, the exception and the yaml. I also provided the
> stacktrace from the following startup exception
>
> ===unit test==============================
>
> @Test
> public void testInvalidColumnNames()
> throws Exception
> {
> final Charset CHARSET = Charset.forName("UTF-8");
>
> int count = 1300000;
> for (int i = 0; i < count; i++)
> {
> List<RowMutation> mutations = new ArrayList<RowMutation>();
> RowMutation m = new RowMutation("Test",
> ByteBuffer.wrap(("rowId").getBytes(CHARSET)));
> // QueryPath qp = new QueryPath("TestCassandra", null,
> ByteBuffer.wrap(("test" + i).getBytes(CHARSET)));
> QueryPath qp = new QueryPath("TestCassandra", null,
> FBUtilities.EMPTY_BYTE_BUFFER);
>
> m.add(qp, ByteBuffer.wrap("test".getBytes(CHARSET)),
> System.currentTimeMillis());
> mutations.add(m);
>
> StorageProxy.mutate(mutations, ConsistencyLevel.ONE);
> }
>
> StorageService.instance.forceTableFlush(CassandraInitializer.KS,
> "TestCassandra");
>
> Thread.sleep(30*1000);
> }
>
>
> ===exception when running test=========================================
>
> 13:54:38 ERROR [PrecompactedRow.<init>] Skipping row
> DecoratedKey(rowId, 726f774964) in
> ./target/test/cassandra/data/Test/TestCassandra-e-4-Data.db
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
> at
> org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:137)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:132)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:312)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> ===yaml file============================================
>
> # Cassandra YAML generated from previous config
> # Configuration wiki: http://wiki.apache.org/cassandra/StorageConfiguration
> authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
> auto_bootstrap: false
> binary_memtable_throughput_in_mb: 256
> cluster_name: CluperUnitTest
> column_index_size_in_kb: 64
> commitlog_directory: ./target/test/cassandra/commitlog
> commitlog_rotation_threshold_in_mb: 128
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 10000
> compaction_thread_priority: 1
> concurrent_reads: 8
> concurrent_writes: 32
> data_file_directories:
> - ./target/test/cassandra/data
> disk_access_mode: auto
> dynamic_snitch: false
> dynamic_snitch_badness_threshold: 0.0
> dynamic_snitch_reset_interval_in_ms: 600000
> dynamic_snitch_update_interval_in_ms: 100
> endpoint_snitch: org.apache.cassandra.locator.RackInferringSnitch
> hinted_handoff_enabled: true
> in_memory_compaction_limit_in_mb: 256
> index_interval: 128
> saved_caches_directory: ./target/test/cassandra/saved_caches
> keyspaces:
> - column_families:
> - column_metadata: []
> column_type: Standard
> compare_with: UTF8Type
> gc_grace_seconds: 864000
> key_cache_save_period_in_seconds: 3600
> keys_cached: 200000.0
> max_compaction_threshold: 32
> memtable_flush_after_mins: 60
> memtable_operations_in_millions: 0.3
> memtable_throughput_in_mb: 64
> min_compaction_threshold: 4
> name: TestCassandra
> read_repair_chance: 1.0
> row_cache_save_period_in_seconds: 0
> rows_cached: 0.0
> name: Test
> replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
> replication_factor: 1
> listen_address: localhost
> partitioner: org.apache.cassandra.dht.OrderPreservingPartitioner
> phi_convict_threshold: 8
> rpc_keepalive: true
> rpc_port: 9161
> rpc_timeout_in_ms: 10000
> seeds:
> - 127.0.0.1
> sliced_buffer_size_in_kb: 64
> snapshot_before_compaction: false
> storage_port: 7012
> thrift_framed_transport_size_in_mb: 15
> thrift_max_message_length_in_mb: 16
>
> ===start up exception after unit test===================================
>
> 14:16:14 ERROR [AbstractCassandraDaemon.activate] Exception encountered
> during startup.
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
> at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
> at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:385)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:395)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
> at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:353)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:283)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:194)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:143)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
> at
> senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217)
> at java.lang.Thread.run(Thread.java:662)
> at
> senselogic.cluper.cassandra.CassandraInitializer$1.run(CassandraInitializer.java:67)
> Exception encountered during startup.
> at java.lang.Thread.run(Thread.java:662)'
>
> ===ls of the commit log directory after the error=================
>
> mikael@mikael2:~/projects/sitevision/trunk/framework/sitevision-framework-core/target/test-classes/target/test/cassandra/commitlog>
> ls -ltr
> total 187936
> -rw-r--r-- 1 mikael wheel 134217730 2011-02-01 14:13
> CommitLog-1296565996491.log
> -rw-r--r-- 1 mikael wheel 28 2011-02-01 14:13
> CommitLog-1296566017932.log.header
> -rw-r--r-- 1 mikael wheel 52 2011-02-01 14:13
> CommitLog-1296565996491.log.header
> -rw-r--r-- 1 mikael wheel 58211952 2011-02-01 14:13
> CommitLog-1296566017932.log
> -rw-r--r-- 1 mikael wheel 0 2011-02-01 14:16
> CommitLog-1296566174499.log
> -rw-r--r-- 1 mikael wheel 20 2011-02-01 14:16
> CommitLog-1296566174499.log.header
>
> --
> Mikael Wikblom
> Software Architect
> SiteVision AB
> 019-217058
> mikael.wikb...@sitevision.se
> http://www.sitevision.se
>
