Quick update, saw the same error on another new node, again the node isn't really misbehaving uptill now.
Thanks On Mon, Jun 8, 2015 at 9:48 PM, Aiman Parvaiz <ai...@flipagram.com> wrote: > Hi everyone > I am running C* 2.0.9 and decided to do a rolling upgrade. Added a node of > C* 2.0.15 in the existing cluster and saw this twice: > > Jun 9 02:27:20 prod-cass23.localdomain cassandra: 2015-06-09 02:27:20,658 > INFO CompactionExecutor:4 CompactionTask.runMayThrow - Compacting > [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-37-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-40-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-42-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-38-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-39-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-44-Data.db')] > > > > Jun 9 02:27:20 prod-cass23.localdomain cassandra: 2015-06-09 02:27:20,669 > ERROR CompactionExecutor:4 CassandraDaemon.uncaughtException - Exception in > thread Thread[CompactionExecutor:4,1,main] > Jun 9 02:27:20 prod-cass23.localdomain > *java.lang.NegativeArraySizeException* > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:335) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:462) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:448) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:432) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableReader.getAncestors(SSTableReader.java:1366) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata.createCollector(SSTableMetadata.java:134) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.createCompactionWriter(CompactionTask.java:316) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:162) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > Jun 9 02:27:20 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > Jun 9 02:27:20 prod-cass23.localdomain at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > Jun 9 02:27:20 prod-cass23.localdomain at > java.util.concurrent.FutureTask.run(FutureTask.java:262) > Jun 9 02:27:20 prod-cass23.localdomain at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > Jun 9 02:27:20 prod-cass23.localdomain at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Jun 9 02:27:20 prod-cass23.localdomain at > java.lang.Thread.run(Thread.java:745) > Jun 9 02:27:47 prod-cass23.localdomain cassandra: 2015-06-09 02:27:47,725 > INFO main StorageService.setMode - JOINING: Starting to bootstrap... > > As you can see this happened first time even before Joining. Second > occasion stack trace: > > Jun 9 02:32:15 prod-cass23.localdomain cassandra: 2015-06-09 02:32:15,097 > ERROR CompactionExecutor:6 CassandraDaemon.uncaughtException - Exception in > thread Thread[CompactionExecutor:6,1,main] > Jun 9 02:32:15 prod-cass23.localdomain > java.lang.NegativeArraySizeException > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:335) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:462) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:448) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:432) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableReader.getAncestors(SSTableReader.java:1366) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.io.sstable.SSTableMetadata.createCollector(SSTableMetadata.java:134) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.createCompactionWriter(CompactionTask.java:316) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:162) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > Jun 9 02:32:15 prod-cass23.localdomain at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > Jun 9 02:32:15 prod-cass23.localdomain at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > Jun 9 02:32:15 prod-cass23.localdomain at > java.util.concurrent.FutureTask.run(FutureTask.java:262) > Jun 9 02:32:15 prod-cass23.localdomain at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > Jun 9 02:32:15 prod-cass23.localdomain at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Jun 9 02:32:15 prod-cass23.localdomain at > java.lang.Thread.run(Thread.java:745) > > The node is not mis behaving as such and I am not seeing any behavior out > of ordinary as of now. Please advice about this error and if possible why > this occurred in first place. Any help is appreciated. > > Thanks > -- Lead Systems Architect 10351 Santa Monica Blvd, Suite 3310 Los Angeles CA 90025
