Re: Data Loss/Missing With Cassandra

2013-06-14 Thread Robert Coli
On Fri, Jun 14, 2013 at 2:05 PM, Tyler Hobbs wrote: > Your data is not marked as deleted. sstableloader will stream all of the > data on that node to all other nodes that need it. http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra Has some details on the difference between

Re: Data Loss/Missing With Cassandra

2013-06-14 Thread Tyler Hobbs
On Mon, Jun 10, 2013 at 1:11 PM, Nimi Wariboko Jr wrote: > > 1.) Seeing as how I only had a single node cluster previously, if I just > `nodetool move 0` the original node that should be an easy fix then? > > You don't need to move the node (and it won't fix everything), just run sstableloader.

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
Thanks for all the advice I really appreciate it. 1.) Seeing as how I only had a single node cluster previously, if I just `nodetool move 0` the original node that should be an easy fix then? 2.) Is my data not marked as deleted? If I use sstableloader to restream the data am I just reorganizi

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Edward Capriolo
This is true unless you have ran cleanup. If you have run 'nodetool cleanup' then there is data loss. On Mon, Jun 10, 2013 at 1:40 PM, Tyler Hobbs wrote: > > On Mon, Jun 10, 2013 at 11:39 AM, Nimi Wariboko Jr < > nimiwaribo...@gmail.com> wrote: > >> How can I recover that data? Can I assume the

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Tyler Hobbs
On Mon, Jun 10, 2013 at 11:39 AM, Nimi Wariboko Jr wrote: > How can I recover that data? Can I assume they are still in the sstables? Yes > Would doing a sstable2json then reading and reinserting be an optimal > solution? You can just use the sstableloader utility to load them directly. It

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Edward Capriolo
To recover it would be to dump everything then re-insert everything. Another option would be to return all nodes to whatever tokens they were before the switches, since the old data is still there. Either way both recovery options are long,painful, and a good amount of manual steps. I would not w

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
How can I recover that data? Can I assume they are still in the sstables? Would doing a sstable2json then reading and reinserting be an optimal solution? On Monday, June 10, 2013 at 9:18 AM, Tyler Hobbs wrote: > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr (mailto:nimiwaribo...@gmail.co

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Edward Capriolo
Although there is a ticket open that WILL start removing data automatically. Which I keep saying is a bad idea. On Mon, Jun 10, 2013 at 12:18 PM, Tyler Hobbs wrote: > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr > wrote: > >> If I had to do a repair after upping the RF, than that is pro

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Tyler Hobbs
On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr wrote: > If I had to do a repair after upping the RF, than that is probably what > caused the data loss. Wish I had been more careful. > > I'm guessing the data is irrevocably lost, I didn't make any any snapshots. > > Would it be possible to figure

Re: Data Loss/Missing With Cassandra

2013-06-09 Thread Nimi Wariboko Jr
If I had to do a repair after upping the RF, than that is probably what caused the data loss. Wish I had been more careful. I'm guessing the data is irrevocably lost, I didn't make any any snapshots. Would it be possible to figure out if only a certain part of the ring was effected? That would

Re: Data Loss/Missing With Cassandra

2013-06-09 Thread Edward Capriolo
Sounds like your cluster got shufflef*cked. You said : "After we had gotten all the data moved over we decided to add 2 more nodes, as well as up the RF to 2." After your raise replication you have to run repair on all nodes. If you did not, and then you proceeded to shuffle you will likely have a

Data Loss/Missing With Cassandra

2013-06-08 Thread Nimi Wariboko Jr
Hi, We are seeing an issue where data that was written to the cluster is no longer accessible after trying to expand the size of the cluster. I will try and provide as much information as possible, I am just starting at with Cassandra and I'm not entirely sure what data is relevant. All Cassa