Thanks all - a little clarification: - The node has fully joined at this point with the duplicates - Cleanup has been run on older nodes - Currently using LCS
Rob - thanks for that, I was wondering whether either of those would successfully deduplicate the data. We were hypothesizing that a decommission would merely stream the duplicates out as well as though they were valid data - is this not the case? After some discussion just now a colleague is suggesting we force them to L0[1] - would you agree this should be equivalent to option 2, albeit with downtime? Cheers, Alain [1] https://issues.apache.org/jira/browse/CASSANDRA-5271 On Tue, Nov 18, 2014 at 12:13 AM, Robert Coli <rc...@eventbrite.com> wrote: > On Mon, Nov 17, 2014 at 12:04 PM, Alain Vandendorpe <al...@tapstream.com> > wrote: > >> With bootstrapping and initial compactions finished that node now has >> what seems to be duplicate data, with almost exactly 2x the expected disk >> usage. CQL returns correct results but we depend on the ability to directly >> read the SSTable files (hence also RF=1.) >> >> Would anyone have suggestions on a good way to resolve this? >> > > (If I understand correctly, the new node is now joined to the cluster; the > below assumes this.) > > ** The simplest, slightly inconvenient, way, which temporarily reduces > capacity : > > 1) nodetool cleanup # on the original node > > This removes obsolete data from the original node, which has moved to the > new node. I mention this in case you did not already do it as part of > joining the new node. There are some edge cases [1] where you will "wake > up" old data if you haven't done this before decommission of the new node. > > 2) nodetool decommission # on the new node > > This moves the data from the new node back onto the old node and removes > the new node from the cluster. > > 3) wipe data from new node # including the system keyspace > > 4) re-bootstrap new node > > ** The second simplest way, which requires using Size Tiered Compaction > Strategy (STS) but does not reduce capacity until step 2) : > > 1) nodetool compact > > This will merge all your duplicates into One Big SSTable. > > 2) if necessary, once RF>1, use sstablesplit [1] (with the node down) to > split up your One Big SSTable. > > If you're not using STS, you can temporarily switch to it, but 2) becomes > less straightforward. > > =Rob > http://twitter.com/rcolidba > [1] https://issues.apache.org/jira/browse/CASSANDRA-7764 > [2] > http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsSSTableSplit.html > >
