What RF were you using and had you been running repair regularly ? Cheers
----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/03/2012, at 5:51 AM, Casey Deccio wrote: > On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio <ca...@deccio.net> wrote: > On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <ca...@deccio.net> wrote: > I recently had to do some shuffling with one of my cassandra nodes because it > was running out of disk space. I did a few things in the process, and I'm > not sure in the end which caused my problem. First I added a second file > path to the data directory in cassandra.yaml. Things still worked fine after > this, as far as I could tell. Shortly after this, however, I took down the > node and rsync'd the data from both data directories, as well as commitlogs, > to an external drive. I then shut down the machine, replaced the hard drives > with bigger drives, and re-installed the OS. I re-created the data > directories, rsync'd the data and commitlogs back over from the external > drive, and started up cassandra, re-adding it to the ring. When it came up, > all of my rows were missing for one columnfamily and nearly all my rows were > missing for another--or at least that's what it looks like, based on walking > the rows. I tried scrubbing each of the nodes. One of them had insufficient > disk space (yes, this seems to be a recurring problem) for scrub, so I did > upgradesstables instead, and that one is still in progress. So far the > scrub/upgradesstables hasn't seemed to help. But in the log messages created > during scrub/upgradesstables it shows realistic numbers (i.e., in terms of > the rows that existed before this ordeal) created in each new sstable. Also, > the loads shown when I run nodetool ring still reflects the numbers with the > complete set of rows. That's encouraging, but I can't seem to access these > phantom rows. Please help! > > > I neglected to mention that I'm running cassandra 1.0.7. > > > Apologies for replying to my own post (again), but here's the follow up. I > decommissioned the newest of the four nodes in the cluster, which was > carrying hardly any load (I'm using ByteOrderedPartitioner), but after I > decommissioned, rows were available again, but only as they were from 10 days > ago. Supercolumns added after that date weren't around. > > Casey
