Hi,
We are setting up a 6-node cassandra cluster within one data center. 3 in
rack1 and the other 3 in rack2. The tokens are assigned alternating
between rack 1 and rack 2. There is one seed node in each rack. Below is
the ring:
r1-node1 DC1 r1 0 (seed)
r2-node1 DC1 r2 28356863910078205288614550619314017621
r1-node2 DC1 r1 56713727820156410577229101238628035242
r2-node2 DC1 r2 85070591730234615865843651857942052863
r1-node3 DC1 r1 113427455640312821154458202477256070484
r2-node3 DC1 r2 141784319550391026443072753096570088105
(seed)
for our schema, we are using NetworkTopologyStrategy, setting DC1
replication factor to 2. We loaded the schema via cassandra-cli on
r1-node1, and cassandra-cli seemed to have succeeded, but we later
discovered that some nodes didn't load the complete keyspace schema
definition.
On these nodes, show keyspaces from cassandra-cli would show the keyspace,
but column families are empty. The system.log file also shows the
following stack trace:
INFO [FlushWriter:4] 2012-05-22 16:56:13,677 Memtable.java (line 307)
Completed flushing
/r1-node2/cassandra-data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6-Data.db
(1428 bytes)
ERROR [MigrationStage:1] 2012-05-22 16:56:13,706
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[MigrationStage:1,5,main]
java.lang.RuntimeException: java.nio.charset.MalformedInputException: Input
length = 1
at
org.apache.cassandra.cql3.ColumnIdentifier.<init>(ColumnIdentifier.java:50)
at
org.apache.cassandra.cql3.CFDefinition.getKeyId(CFDefinition.java:125)
at
org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:59)
at
org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1278)
at
org.apache.cassandra.config.CFMetaData.keyAlias(CFMetaData.java:221)
at
org.apache.cassandra.config.CFMetaData.fromSchemaNoColumns(CFMetaData.java:1162)
at
org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1190)
at
org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:291)
at
org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:395)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
at
org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
at
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:163)
at
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:120)
at
org.apache.cassandra.cql3.ColumnIdentifier.<init>(ColumnIdentifier.java:46)
... 18 more
Has anyone seen this before? Thanks.
-- Y.
