So you have deleted the partition. Do not delete the sstables directly. By default cassandra will keep the tombstones untouched for 10 days. Once 10 days have passed (should be done now since your message was on august 12) a compaction is needed to actually reclaim the data.
You could force a compaction manually but before advising you to do so, could you tell us what compaction strategy you are using for this table ? Regards, Leo On Mon, Aug 12, 2019 at 6:13 PM Gabriel Giussi <gabrielgiu...@gmail.com> wrote: > I've found a huge partion (~9GB) in my cassandra cluster because I'm > loosing 3 nodes recurrently due to OutOfMemoryError > >> ERROR [SharedPool-Worker-12] 2019-08-12 11:07:45,735 >> JVMStabilityInspector.java:140 - JVM state determined to be unstable. >> Exiting forcefully due to: >> java.lang.OutOfMemoryError: Java heap space >> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_151] >> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_151] >> at >> org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:373) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:267) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:193) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:109) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:97) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:301) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:321) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> ~[na:1.8.0_151] >> at >> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) >> ~[apache-cassandra-3.0.13.jar:3.0.13] >> at >> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) >> [apache-cassandra-3.0.13.jar:3.0.13] >> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >> [apache-cassandra-3.0.13.jar:3.0.13] >> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] >> > > From the stacktrace I assume that some client is try to read that > partition (ReadResponse) so I may filter requests to this specific > partition as a quick solution but I think the compaction will never be able > to remove this partition (I already executed a DELETE). > What can I do to delete this partition? May I delete the sstable directly? > Or should I upgrade the node and give more heap to cassandra? > > Thanks. >
