Cool, thanks for the trick. Dean On 2/28/13 5:55 PM, "Erik Forkalsud" <eforkals...@cj.com> wrote:
> >Have you tried to (via jmx) call >org.apache.cassandra.db.CompactionManager.forceUserDefinedCompaction() >and give it the name of your SSTable file. > >It's a trick I use to aggressively get rid of expired data, i.e. if I >have a column family where all data is written with a TTL of 30 days, >any SSTable files with last modified time of more than 30 days ago will >have only expired data, so I call the above function to compact those >files one by one. In your case it sounds like it's not expired data, >but data that belongs on other nodes that you want to get rid of. I'm >not sure if compaction will drop data that doesn't fall within the nodes >key range, but if it does this method should have the effect you're after. > > >- Erik - > > >On 02/27/2013 08:51 PM, Hiller, Dean wrote: >> Okay, we had 6 nodes of 130Gig and it was slowly increasing. Through >>our operations to modify bloomfilter fp chance, we screwed something up >>as trying to relieve memory pressures was tough. Anyways, somehow, this >>caused nodes 1, 2, and 3 to jump to around 200Gig and our incoming data >>stream is completely constant at around 260 points/second. >> >> Sooo, we know this dangling data(around 60Gigs) is in one single column >>family. Node 1, 2, and 3 is for the first token range according to >>ringdescribe. It is almost like the issue is now replicated to the >>other two nodes. Is there any way we can go about debugging this and >>release the 60 gigs of disk space? >> >> Also, the upgradesstables when memory is already close to max is not >>working too well. Can we do this instead(ie. Is it safe?)? >> >> 1. Bring down the node >> 2. Move all the *Index.db files to another directory >> 3. Start the node and run upgradesstables >> >> We know this relieves a ton of memory out of the gate for us. We are >>trying to get memory back down by a gig, then upgrade to 1.2.2 and >>switch to leveled compaction as we have ZERO I/o really going on most of >>the time and really just have this bad bad memory bottleneck(iostat >>shows nothing typically as we are bottlenecked by memory). >> >> Thanks, >> Dean >
