i see leveled compaction used, if it's last, it will have to stay until
next level compaction happens, then will be gone, right ?
On Thu, Feb 1, 2018 at 2:33 AM, Bo Finnerup Madsen <bo.gunder...@gmail.com>
wrote:
> Hi,
>
> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
> generally runs fine, but we have one table that are causing OOMs because an
> enormous amount of tombstones.
> Looking at the data in the table (sstable2json), the first of the
> tombstones are almost a year old. The table was initially created with a
> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
> I have run a full repair of the table across all nodes. I have forced
> several major compactions of the table by using "nodetool compact", and
> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>
> What could cause cassandra to keep these tombstones?
>
> sstable2json:
> {"key": "foo",
> "cells": [["0000082f-25ef-4324-bb8a-8cf013c823c1:_","0000082f-
> 25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
> ["000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","000010f3-
> c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
> ["00001d7a-ce95-4c74-b67e-f8cdffec4f85:_","00001d7a-
> ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
> ["00001dd3-ae22-4f6e-944a-8cfa147cde68:_","00001dd3-
> ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
> ["000022cc-d69c-4596-89e5-3e976c0cb9a8:_","000022cc-
> d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
> ["00002777-4b1a-4267-8efc-c43054e63170:_","00002777-
> 4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
> ["000061e8-f48b-4484-96f1-f8b6a3ed8f9f:_","000061e8-
> f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
> ["000063da-f165-449b-b65d-2b7869368734:_","000063da-
> f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
> ["0000656f-f8b5-472b-93ed-1a893002f027:_","0000656f-
> f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
> ...
> {"key": "bar",
> "metadata": {"deletionInfo": {"markedForDeleteAt":1517402198585982,"
> localDeletionTime":1517402198}},
> "cells": [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-
> ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-
> 7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-
> aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
> ....
>
> sstablemetadata:
> stablemetadata /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1
> c6/ddp-yyy-ka-2741-Data.db
> SSTable: /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1
> c6/ddp-yyy-ka-2741
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.100000
> Minimum timestamp: 1488976211688000
> Maximum timestamp: 1517468644066000
> SSTable max local deletion time: 2147483647 <(214)%20748-3647>
> Compression ratio: 0.5121956624389545
> Estimated droppable tombstones: 18.00161766553587
> SSTable Level: 0
> Repaired at: 0
> ReplayPosition(segmentId=1517168739626, position=22690189)
> Estimated tombstone drop times:%n
> 1488976211: 1
> 1489906506: 4706
> 1490174752: 6111
> 1490449759: 6554
> 1490735410: 6559
> 1491016789: 6369
> 1491347982: 10216
> 1491680214: 13502
> ...
>
> desc:
> CREATE TABLE xxx.yyy (
> ti text,
> uuid text,
> json_data text,
> PRIMARY KEY (ti, uuid)
> ) WITH CLUSTERING ORDER BY (uuid ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 'org.apache.cassandra.io.
> compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
>
> jmx props(picture):
> [image: image.png]
>
