[
https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481209#comment-13481209
]
Joe Fox commented on CASSANDRA-4832:
------------------------------------
I've run into the same symptoms as Chris, but while attempting to migrate data
between a 1.1.1 cluster and 1.1.6.
Whilst using sstableloader to import the data, nodes would hit certain column
indexes and stop processing further requests once the assertion was thrown -
restarting the node would almost immediately throw the assertion again, and the
node would just fail to rejoin the ring.
tpstats would show active flushwriter tasks but no further node activity.
We had to work around the issue by:
1. Removing all indexes from our target cluster schema
2. Importing the data via sstableloader
3. Scanning through relevant column families and inserting data into any empty
indexed columns
4. Re-applying the indexes to the target cluster schema
Only then was the migration successful.
I'll also note that attempting to apply an index to a column which has null
data will also throw the cluster out of sync as the nodes which throw the
assertion fail to migrate their schemas properly
INFO [Creating index: Transactions.TransactionsCountryCode] 2012-10-21
07:45:25,978 ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512
serialized/live bytes, 762 ops)
INFO [Creating index: Transactions.TransactionsStatus] 2012-10-21 07:45:25,980
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live
bytes, 762 ops)
INFO [FlushWriter:1] 2012-10-21 07:45:25,987 Memtable.java (line 264) Writing
Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512
serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,004 Memtable.java (line 264) Writing
Memtable-Transactions.TransactionsStatus@673679943(38862/125966 serialized/live
bytes, 762 ops)
ERROR [FlushWriter:1] 2012-10-21 07:45:26,004 AbstractCassandraDaemon.java
(line 135) Exception in thread Thread[FlushWriter:1,5,main]
java.lang.AssertionError: Keys must not be empty
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [FlushWriter:2] 2012-10-21 07:45:26,049 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsStatus-hf-2-Data.db
(35259 bytes) for commitlog position ReplayPosition(segmentId=1350805525722,
position=0)
INFO [Creating index: Transactions.TransactionsLastUpdateDate] 2012-10-21
07:45:26,313 ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277
serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,314 Memtable.java (line 264) Writing
Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277
serialized/live bytes, 762 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:26,743 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsLastUpdateDate-hf-2-Data.db
(37024 bytes) for commitlog position ReplayPosition(segmentId=1350805525722,
position=0)
INFO [main] 2012-10-21 07:45:27,052 CommitLogReplayer.java (line 272) Finished
reading /var/lib/cassandra/commitlog/CommitLog-1350768744805.log
INFO [main] 2012-10-21 07:45:27,054 ColumnFamilyStore.java (line 659)
Enqueuing flush of Memtable-Versions@1851630436(83/103 serialized/live bytes, 3
ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,054 Memtable.java (line 264) Writing
Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
INFO [FlushWriter:2] 2012-10-21 07:45:27,061 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/Versions/system-Versions-hf-1-Data.db (247
bytes) for commitlog position ReplayPosition(segmentId=1350805525722,
position=0)
> AssertionError: keys must not be empty
> --------------------------------------
>
> Key: CASSANDRA-4832
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Debian 6.0.5
> Reporter: Tristan Seligmann
> Assignee: Tristan Seligmann
> Priority: Minor
> Labels: indexing
> Fix For: 1.1.7
>
> Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
> INFO 07:08:32,104 Compacting
> [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'),
>
> SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
> at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged
> during a repair but I can't reproduce the error 100% reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
