Also, to be sure, were you using the 1.0.0 final or some RC when getting this exception?
On Fri, Oct 21, 2011 at 8:24 PM, Sylvain Lebresne <sylv...@datastax.com> wrote: > Would you have the full log for one of those node leading to the exception > that you could share? Not sure that'll help but who knows. > > -- > Sylvain > > On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <rames...@gmail.com> wrote: >> i am using size based compaction ( the default one ). Also this is on linux. >> >> thanks >> Ramesh >> >> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sylv...@datastax.com> >> wrote: >>> I believe this is the same as >>> https://issues.apache.org/jira/browse/CASSANDRA-3306. >>> >>> The initial reporter only got this exception with leveled compaction, >>> is it what you are >>> using too (to help narrow it down)? Also, are you using windows by any >>> chance? >>> >>> -- >>> Sylvain >>> >>> >>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <rames...@gmail.com> >>> wrote: >>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this >>>> exception quite often. Any idea how to debug this issue? >>>> >>>> >>>> java.lang.IllegalArgumentException: Illegal Capacity: -2 >>>> at java.util.ArrayList.<init>(ArrayList.java:110) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) >>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) >>>> at >>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> >>>> >>>> >>>> few lines before this error >>>> >>>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line >>>> 237) Writing Memtable-participants@1907757288(6777526/153164388 >>>> serialized/live bytes, 199339 ops) >>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line >>>> 488) Discarding obsolete commit >>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log) >>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line >>>> 488) Discarding obsolete commit >>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line >>>> 273) Completed flushing >>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line >>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live >>>> bytes, 426 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line >>>> 273) Completed flushing >>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line >>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008 >>>> serialized/live bytes, 109823 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line >>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db >>>> (8125165 bytes) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line >>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live >>>> bytes, 611301 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line >>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db >>>> (48225128 bytes) >>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line >>>> 488) Discarding obsolete commit >>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log) >>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line >>>> 488) Discarding obsolete commit >>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log) >>>> INFO [MutationStage:88] 2011-10-20 10:52:29,905 >>>> ColumnFamilyStore.java (line 664) Enqueuing flush of >>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007 >>>> ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line >>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live >>>> bytes, 5007 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line >>>> 273) Completed flushing >>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes) >>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132 >>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >>>> Thread[CompactionExecutor:538,1,main] >>>> >>>> >>>> >>>> Another one >>>> >>>> >>>> >>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line >>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live >>>> bytes, 1111 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line >>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db >>>> (142584 bytes) >>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778 >>>> CompactionTask.java (line 119) Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), >>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')] >>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935 >>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >>>> Thread[CompactionExecutor:544,1,main] >>>> java.lang.AssertionError >>>> at >>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) >>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) >>>> at >>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> INFO [MutationStage:89] 2011-10-20 10:52:46,599 >>>> ColumnFamilyStore.java (line 664) Enqueuing flush of >>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624 >>>> ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line >>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live >>>> bytes, 2624 ops) >>>> >>>> >>>> >>>> >>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line >>>> 488) Discarding obsolete commit >>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log) >>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058 >>>> CompactionTask.java (line 119) Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), >>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')] >>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552 >>>> CompactionTask.java (line 222) Compacted to >>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to >>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS. >>>> Time: 494ms. >>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901 >>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >>>> Thread[CompactionExecutor:539,1,main] >>>> java.lang.AssertionError >>>> at >>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) >>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) >>>> at >>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983 >>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >>>> Thread[CompactionExecutor:542,1,main] >>>> java.lang.IllegalArgumentException: Illegal Capacity: -3 >>>> at java.util.ArrayList.<init>(ArrayList.java:110) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) >>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) >>>> at >>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618 >>>> CompactionTask.java (line 222) Compacted to >>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308 >>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at >>>> 3.594734MBPS. Time: 14,417ms. >>>> INFO [MutationStage:75] 2011-10-20 10:52:53,877 >>>> ColumnFamilyStore.java (line 664) Enqueuing flush of >>>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes, >>>> 4232 ops) >>>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line >>>> 237) Writing Memtable-participants@1814149120(143888/3251705 >>>> serialized/live bytes, 4232 ops) >>>> >>>> >>>> >>>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313 >>>> CompactionTask.java (line 119) Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'), >>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'), >>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'), >>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')] >>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532 >>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >>>> Thread[CompactionExecutor:542,1,main] >>>> java.lang.IllegalArgumentException: Illegal Capacity: -2 >>>> at java.util.ArrayList.<init>(ArrayList.java:110) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) >>>> at >>>> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) >>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) >>>> at >>>> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) >>>> at >>>> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787 >>>> CompactionTask.java (line 222) Compacted to >>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to >>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS. >>>> Time: 605ms. >>>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806 >>>> CompactionTask.java (line 222) Compacted to >>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to >>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS. >>>> Time: 748ms. >>>> >>>> >>>> thanks >>>> Ramesh >>>> >>> >> >
