Re: Weird problem with empty CF

2011-10-05 Thread aaron morton
No. It's generally only an issue with heavy delete workloads, and it's sometimes possible to design around it. cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 5/10/2011, at 1:18 PM, Daning wrote: > Thanks. Do you have plan t

Re: Weird problem with empty CF

2011-10-04 Thread Daning
Thanks. Do you have plan to improve this? I think tombstone should be separated with live data since it serves different purpose, built in separate SSTable or indexed differently. It is pretty costly to do filtering while reading. Daning On 10/04/2011 01:34 PM, aaron morton wrote: I would n

Re: Weird problem with empty CF

2011-10-04 Thread aaron morton
I would not get gc_grace seconds to 0, set to to something small. gc_grace_seconds or ttl is only the minimum amount of time the column will stay in the data files. The columns are only purged when compaction runs some time after that timespan has ended. If you are seeing issues where a heavy

Re: Weird problem with empty CF

2011-10-04 Thread Daning
Thanks Aaron. How about I set the gc_grace_seconds to 0 or like 2 hours? I like to clean up tomebstone sooner, I don't care losing some data and all my columns have ttl. If one node is down longer than gc_grace_seconds, and I got tombstone removed, once the node is up, from my understanding d

Re: Weird problem with empty CF

2011-10-04 Thread aaron morton
Yes that's the slice query skipping past the tombstone columns. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 4/10/2011, at 4:24 PM, Daning Wang wrote: > Lots of SliceQueryFilter in the log, is that handling tombstone? > > DEB

Re: Weird problem with empty CF

2011-10-03 Thread Daning Wang
Lots of SliceQueryFilter in the log, is that handling tombstone? DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000 DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1

Re: Weird problem with empty CF

2011-09-29 Thread aaron morton
As with any situation involving the un-dead, it really is the number of Zombies, Mummies or Vampires that is the concern. If you delete data there will always be tombstones. If you have a delete heavy workload there will be more tombstones. This is why implementing a queue with cassandra is a

Re: Weird problem with empty CF

2011-09-29 Thread Daning Wang
Jonathan/Aaron, Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will happen. Is there a way to purge tombstones at anytime? because if tombstones affect performance, we want them to be purged right away, not after GCGracePeriod. We know all the nodes are up, and we can do

Re: Weird problem with empty CF

2011-09-28 Thread aaron morton
if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice" Minor (automatic) compaction will, over time, purge the tombstones. Until then reads mu

Re: Weird problem with empty CF

2011-09-28 Thread Jonathan Ellis
Sounds like you have non-expired tombstones. http://wiki.apache.org/cassandra/DistributedDeletes On Wed, Sep 28, 2011 at 12:35 PM, Daning wrote: > I have an app polling a few CFs (select first N * from CF), there were data > in CFs but later were deleted so CFs were empty for a long time. I foun

Weird problem with empty CF

2011-09-28 Thread Daning
I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I h