Thank you all for your help. I was able to get rid of zombies (at least end users not reporting that anymore) using nodetool cleanup. And old SSTables were indeed unable to merge with each other because of repairedAt > 0, so cassandra stop + sstablerepairedset + cassandra start in rolling manner did fix the issue.
Thanks. On Fri, Dec 15, 2017 at 10:47 PM, kurt greaves <k...@instaclustr.com> wrote: > X==5. I was meant to fill that in... > > On 16 Dec. 2017 07:46, "kurt greaves" <k...@instaclustr.com> wrote: > >> Yep, if you don't run cleanup on all nodes (except new node) after step >> x, when you decommissioned node 4 and 5 later on, their tokens will be >> reclaimed by the previous owner. Suddenly the data in those SSTables is now >> live again because the token ownership has changed and any data in those >> SStables will be returned. >> >> Remember new nodes only add tokens to the ring, they don't affect other >> nodes tokens, so if you remove those tokens everything goes back to how it >> was before those nodes were added. >> >> Adding a maker would be incredibly complicated. Plugs not really fit the >> design of Cassandra. Here it's probably much easier to just follow >> recommended procedure when adding and removing nodes. >> >> On 16 Dec. 2017 01:37, "Python_Max" <python....@gmail.com> wrote: >> >> Hello, Jeff. >> >> >> Using your hint I was able to reproduce my situation on 5 VMs. >> Simplified steps are: >> 1) set up 3-node cluster >> 2) create keyspace with RF=3 and table with gc_grace_seconds=60, >> compaction_interval=10 and unchecked_tombstone_compaction=true (to force >> compaction later) >> 3) insert 10..20 records with different partition and clustering keys >> (consistency 'all') >> 4) 'nodetool flush' on all 3 nodes >> 5) add 4th node, add 5th node >> 6) using 'nodetool getendpoints' find key that moved to both 4th and 5th >> node >> 7) delete that record from table (consistency 'all') >> 8) 'nodetool flush' on all 5 nodes, wait gc_grace_seconds, 'nodetool >> compact' on nodes which responsible for that key, check that key and >> tombstone gone using sstabledump >> 9) decommission 5th node, decommission 4th node >> 10) select data from table where key=key (consistency quorum) >> >> And the row is here. >> >> It sounds like bug in cassandra but since it is documented here >> https://docs.datastax.com/en/cassandra/3.0/cassandra/operati >> ons/opsAddNodeToCluster.html I suppose this counts as feature. It would >> be better when data which stays in sstable after new node added would have >> some marker and never returned as result to select query. >> >> Thank you very much, Jeff, for pointing me in right direction. >> >> >> On 13.12.17 18:43, Jeff Jirsa wrote: >> >>> Did you run cleanup before you shrank the cluster? >>> >>> >> -- >> >> Best Regards, >> Python_Max. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> -- Best regards, Python_Max.
