Did you run cleanup before you shrank the cluster? -- Jeff Jirsa
> On Dec 13, 2017, at 4:49 AM, Python_Max <python....@gmail.com> wrote: > > Hello. > > I have a situation similar to > https://issues.apache.org/jira/browse/CASSANDRA-13153 except mine cassandra > is 3.11.1 (that issue should be fixed according to jira). > > Cluster consist of 40 nodes which I have to shrink to 25 more powerful nodes > moving less powerful out from the cluster. > > SizeTieredCompactionStrategy is used for target table with default compaction > options and default gc_grace_seconds (10 days). > > Full sequential repair executed every day using > https://github.com/thelastpickle/cassandra-reaper > > Before reaper was involved the default (incremental) 'nodetool repair' was > used. > > > There are couple of problems that I observe. > > 1. Compaction. > > There are rows in target table which deleted long time ago (gc_grace_seconds > passed) but they are not compacting. I tried 'nodetool compact' and 'nodetool > repair -full -seq' with same outcome: sstables recreated but that rows still > there (I used sstabledump to detect this state). > > Some of that rows have tombstone and data in wrong order: the data located at > more recent sstable by index but relative tombstone is located in previous > sstable (shouldn't it be opposite?). CQL select does not return that rows > (correct behaviour) until node decommission. > > sstables with non-compacted rows have very old repairedAt value (about 10 > days before the first deleted row in sstable which should have been compacted > long time ago). > > > 2. Streaming. > > When moving the node out of cluster 'nodetool decommission' is used. After > streaming complete some old rows that was not compacted earlier is back to > life in shrinked cluster. > > CQL select does return that rows as alive until running full sequential > repair using cassandra-reaper. > > As workaround I tried to shut down a node and using 'nodetool removenode' in > case the node itself is streaming wrong data on decommission but that did not > work either (deleted data is back to life). > > > Is this a known issue? > > > PS: I have not tried 'nodetool scrub' yet nor dropping repairedAt for > affected sstables. > > > -- > > Best Regards, > Python_Max. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
