[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17945849#comment-17945849
]
Chris Miller commented on CASSANDRA-16764:
------------------------------------------
Hi [~brandon.williams], we're also seeing this behaviour in cassandra 4.1.6.
Were any fixes ever developed for this? Any thoughts how to resolve when the CF
is using STCS?
Thanks,
Chris.
Here;s the stack trace:
error: null
-- StackTrace --
java.nio.BufferOverflowException
at
org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:136)
at
org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:154)
at
org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:161)
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:121)
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:100)
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:72)
at
org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:63)
at
org.apache.cassandra.db.marshal.ByteArrayAccessor.write(ByteArrayAccessor.java:40)
at
org.apache.cassandra.db.marshal.ValueAccessor.writeWithVIntLength(ValueAccessor.java:166)
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:481)
at
org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:397)
at
org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:133)
at
org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:339)
at
org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:110)
at
org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:90)
at
org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:228)
at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:276)
at
org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:122)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:222)
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:134)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:205)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
at
org.apache.cassandra.db.compaction.CompactionManager$9.runMayThrow(CompactionManager.java:1001)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
> Compaction repeatedly fails validateReallocation exception
> ----------------------------------------------------------
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
> Reporter: Richard Hesse
> Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:272)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_292]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_292]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> [apache-cassandra-3.11.10.jar:3.11.10]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
> {code}
> Watching the compaction progress, it looks like it makes it through, dies,
> then starts over again. This process repeats forever.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
