We did start with a 3 node cluster and a RF of 3, then added another 3 nodes and again another 3 nodes. So it is a good guess :) But I have run both repair and cleanup against the table on all nodes, would that not have removed any stray partitions?
tor. 1. feb. 2018 kl. 22.31 skrev Steinmaurer, Thomas < thomas.steinmau...@dynatrace.com>: > Did you started with a 9 node cluster from the beginning or did you extend > / scale out your cluster (with vnodes) beyond the replication factor? > > > > If later applies and if you are deleting by explicit deletes and not via > TTL, then nodes might not see the deletes anymore, as a node might not own > the partition anymore after a topology change (e.g. scale out beyond the > keyspace RF). > > > > Just a very wild guess. > > > > Thomas > > > > *From:* Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com] > *Sent:* Donnerstag, 01. Februar 2018 22:14 > > > *To:* user@cassandra.apache.org > *Subject:* Re: Old tombstones not being cleaned up > > > > We do not use TTL anywhere...records are inserted and deleted "manually" > by our software. > > tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad <j...@jonhaddad.com>: > > Changing the defaul TTL doesn’t change the TTL on the existing data, only > new data. It’s only set if you don’t supply one yourself. > > > > On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen < > bo.gunder...@gmail.com> wrote: > > Hi, > > > > We are running a small 9 node Cassandra v2.1.17 cluster. The cluster > generally runs fine, but we have one table that are causing OOMs because an > enormous amount of tombstones. > > Looking at the data in the table (sstable2json), the first of the > tombstones are almost a year old. The table was initially created with a > gc_grace_period of 10 days, but I have now lowered it to 1 hour. > > I have run a full repair of the table across all nodes. I have forced > several major compactions of the table by using "nodetool compact", and > also tried to switch from LeveledCompaction to SizeTierCompaction and back. > > > > What could cause cassandra to keep these tombstones? > > > > sstable2json: > > {"key": "foo", > > "cells": > [["0000082f-25ef-4324-bb8a-8cf013c823c1:_","0000082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135], > > > > ["000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731], > > > > ["00001d7a-ce95-4c74-b67e-f8cdffec4f85:_","00001d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102], > > > > ["00001dd3-ae22-4f6e-944a-8cfa147cde68:_","00001dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006], > > > > ["000022cc-d69c-4596-89e5-3e976c0cb9a8:_","000022cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448], > > > > ["00002777-4b1a-4267-8efc-c43054e63170:_","00002777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691], > > > > ["000061e8-f48b-4484-96f1-f8b6a3ed8f9f:_","000061e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300], > > > > ["000063da-f165-449b-b65d-2b7869368734:_","000063da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634], > > > > ["0000656f-f8b5-472b-93ed-1a893002f027:_","0000656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716], > > ... > > {"key": "bar", > > "metadata": {"deletionInfo": > {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}}, > > "cells": > [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965], > > > > ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186], > > > > ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840], > > .... > > > > sstablemetadata: > > stablemetadata > /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db > > SSTable: > /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741 > > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > > Bloom Filter FP chance: 0.100000 > > Minimum timestamp: 1488976211688000 > > Maximum timestamp: 1517468644066000 > > SSTable max local deletion time: 2147483647 > > Compression ratio: 0.5121956624389545 > > Estimated droppable tombstones: 18.00161766553587 > > SSTable Level: 0 > > Repaired at: 0 > > ReplayPosition(segmentId=1517168739626, position=22690189 > <22%2069%2001%2089>) > > Estimated tombstone drop times:%n > > 1488976211: 1 > > 1489906506: 4706 > > 1490174752: 6111 > > 1490449759: 6554 > > 1490735410: 6559 > > 1491016789: 6369 > > 1491347982: 10216 > > 1491680214: 13502 > > ... > > > > desc: > > CREATE TABLE xxx.yyy ( > > ti text, > > uuid text, > > json_data text, > > PRIMARY KEY (ti, uuid) > > ) WITH CLUSTERING ORDER BY (uuid ASC) > > AND bloom_filter_fp_chance = 0.1 > > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > > AND comment = '' > > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > > AND dclocal_read_repair_chance = 0.1 > > AND default_time_to_live = 0 > > AND gc_grace_seconds = 3600 > > AND max_index_interval = 2048 > > AND memtable_flush_period_in_ms = 0 > > AND min_index_interval = 128 > > AND read_repair_chance = 0.0 > > AND speculative_retry = '99.0PERCENTILE'; > > > > jmx props(picture): > > [image: image001.png] > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 > <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> >
