Hi Max, I don't know if it's related to your issue but on a side note, if you decide to use Reaper (and use full repairs, not incremental ones), but mix that with "nodetool repair", you'll end up with 2 pools of SSTables that cannot get compacted together. Reaper uses subrange repair which doesn't mark SSTables are repaired (no anticompaction is performed, repairedAt remains at 0), while using nodetool in full and incremental modes will perform anticompaction.
SSTables with repairedAt > 0 cannot be compacted with SSTables with repairedAt = 0. Bottom line is that if you want your SSTables to be compacted together naturally, you have to run repairs either exclusively through Reaper or exclusively through nodetool. If you decide to use Reaper exclusively, you have to revert the repairedAt value to 0 for all sstables on all nodes, using sstablerepairedset <https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSSTableRepairedSet.html> . Cheers, On Fri, Dec 15, 2017 at 4:57 PM Jeff Jirsa <jji...@gmail.com> wrote: > The generation (integer id in file names) doesn’t matter for ordering like > this > > It matters in schema tables for addition of new columns/types, but it’s > irrelevant for normal tables - you could do a user defined compaction on > 31384 right now and it’d be rewritten as-is (minus purgable data) with the > new highest generation, even though it’s all old data. > > > -- > Jeff Jirsa > > > On Dec 15, 2017, at 6:55 AM, Python_Max <python....@gmail.com> wrote: > > Hi, Kurt. > > Thank you for response. > > > Repairs are marked as 'done' without errors in reaper history. > > Example of 'wrong order': > > * file mc-31384-big-Data.db contains tombstone: > > { > "type" : "row", > "position" : 7782, > "clustering" : [ "9adab970-b46d-11e7-a5cd-a1ba8cfc1426" ], > "deletion_info" : { "marked_deleted" : > "2017-10-28T04:51:20.589394Z", "local_delete_time" : "2017-10-28T04:51:20Z" > }, > "cells" : [ ] > } > > * file mc-31389-big-Data.db contains data: > > { > "type" : "row", > "position" : 81317, > "clustering" : [ "9adab970-b46d-11e7-a5cd-a1ba8cfc1426" ], > "liveness_info" : { "tstamp" : "2017-10-19T01:34:10.055389Z" }, > "cells" : [...] > } > Index 31384 is less than 31389 but I'm not sure whether it matters at all. > > I assume that data and tombsones are not compacting due to another reason: > the tokens are not owned by that node anymore and the only way to purge > such keys is 'nodetool cleanup', isn't it? > > > On 14.12.17 16:14, kurt greaves wrote: > > Are you positive your repairs are completing successfully? Can you send > through an example of the data in the wrong order? What you're saying > certainly shouldn't happen, but there's a lot of room for mistakes. > > On 14 Dec. 2017 20:13, "Python_Max" <python....@gmail.com> wrote: > >> Thank you for reply. >> >> No, I did not execute 'nodetool cleanup'. Documentation >> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRemoveNode.html >> does not mention that cleanup is required. >> >> Do yo think that extra data which node is not responsible for can lead to >> zombie data? >> >> >> On 13.12.17 18:43, Jeff Jirsa wrote: >> >>> Did you run cleanup before you shrank the cluster? >>> >>> >> -- >> >> Best Regards, >> Python_Max. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> happen > > > -- > > Best Regards, > Python_Max. > > -- ----------------- Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com
