Here's what I ended up doing on a test cluster. It seemed to work well. I'm running a full repair on the production cluster, probably over the weekend, then I'll have a go at the test cluster again and go for broke.
# sync to temporary directory on original volume rsync -azvuiP /var/data/cassandra_data2/ /var/data/cassandra/data2/ # check "before" size of data directory du -sh /var/data/cassandra/data # compare sizes du -sh /var/data/cassandra_data2 && du -sh /var/data/cassandra/data2 service cassandra stop # sync anything that changed before stop/drain completed rsync -azvuiP /var/data/cassandra_data2/ /var/data/cassandra/data2/ # compare sizes du -sh /var/data/cassandra_data2 && du -sh /var/data/cassandra/data2 # edit /usr/local/cassandra/conf/cassandra.yaml: # - remove /var/data/cassandra_data2 from data_file_directories # sync files into real data directory rsync -azvuiP /var/data/cassandra/data2/ /var/data/cassandra/data/ # check "after" size of data directory (should be size of /var/data/cassandra_data2 plus "before" size) du -sh /var/data/cassandra/data # remove temporary directory rm -Rf /var/data/cassandra/data2 # unmount second volume umount /dev/xvdf # In AWS console: # - detach sdf volume # - delete volume # remove mount directory rm -Rf /var/data/cassandra_data2/ # restart cassandra service cassandra start # run repair /usr/local/cassandra/bin/nodetool repair -pr On Thu, Feb 18, 2016 at 3:12 PM, Branton Davis <branton.da...@spanning.com> wrote: > Alain, thanks for sharing! I'm confused why you do so many repetitive > rsyncs. Just being cautious or is there another reason? Also, why do you > have --delete-before when you're copying data to a temp (assumed empty) > directory? > > On Thu, Feb 18, 2016 at 4:12 AM, Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> I did the process a few weeks ago and ended up writing a runbook and a >> script. I have anonymised and share it fwiw. >> >> https://github.com/arodrime/cassandra-tools/tree/master/remove_disk >> >> It is basic bash. I tried to have the shortest down time possible, making >> this a bit more complex, but it allows you to do a lot in parallel and just >> do a fast operation sequentially, reducing overall operation time. >> >> This worked fine for me, yet I might have make some errors while making >> it configurable though variables. Be sure to be around if you decide to run >> this. Also I automated this more by using knife (Chef), I hate to repeat >> ops, this is something you might want to consider. >> >> Hope this is useful, >> >> C*heers, >> ----------------- >> Alain Rodriguez >> France >> >> The Last Pickle >> http://www.thelastpickle.com >> >> 2016-02-18 8:28 GMT+01:00 Anishek Agarwal <anis...@gmail.com>: >> >>> Hey Branton, >>> >>> Please do let us know if you face any problems doing this. >>> >>> Thanks >>> anishek >>> >>> On Thu, Feb 18, 2016 at 3:33 AM, Branton Davis < >>> branton.da...@spanning.com> wrote: >>> >>>> We're about to do the same thing. It shouldn't be necessary to shut >>>> down the entire cluster, right? >>>> >>>> On Wed, Feb 17, 2016 at 12:45 PM, Robert Coli <rc...@eventbrite.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Feb 16, 2016 at 11:29 PM, Anishek Agarwal <anis...@gmail.com> >>>>> wrote: >>>>>> >>>>>> To accomplish this can I just copy the data from disk1 to disk2 with >>>>>> in the relevant cassandra home location folders, change the cassanda.yaml >>>>>> configuration and restart the node. before starting i will shutdown the >>>>>> cluster. >>>>>> >>>>> >>>>> Yes. >>>>> >>>>> =Rob >>>>> >>>>> >>>> >>>> >>> >> >
