On Fri, Nov 18, 2011 at 6:31 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > On Fri, Nov 18, 2011 at 7:47 AM, Sylvain Lebresne <sylv...@datastax.com> > wrote: >> On Fri, Nov 18, 2011 at 4:23 PM, Mohit Anchlia <mohitanch...@gmail.com> >> wrote: >>> On Fri, Nov 18, 2011 at 6:39 AM, Sylvain Lebresne <sylv...@datastax.com> >>> wrote: >>>> On Fri, Nov 18, 2011 at 1:53 AM, Todd Burruss <bburr...@expedia.com> wrote: >>>>> I'm using cassandra 1.0. Been doing some testing on using cass's cache. >>>>> When I turn it on (using the CLI) I see ParNew jump from 3-4ms to >>>>> 200-300ms. This really screws with response times, which jump from >>>>> ~25-30ms >>>>> to 1300+ms. I've increase new gen and that helps, but still this is >>>>> suprising to me, especially since 1.0 defaults to the >>>>> SerializingCacheProvider – off heap. >>>>> The interesting tid bit is that I have wide rows. 70k+ columns per row, >>>>> ~50 >>>>> bytes per column value. The cache only must be about 400 rows to catch >>>>> all >>>>> the data per node and JMX is reporting 100% cache hits. Nodetool ring >>>>> reports < 2gb per node, my heap is 6gb and total RAM is 16gb. >>>>> Thoughts? >>>> >>>> You're problem is the mix of wide rows and the serializing cache. >>>> What happens with the serializing cache is that our data is stored >>>> out of the heap. But that means that for each read to a row, we >>>> 'deserialize' the row for the out-of-heap memory into the heap to >>>> return it. The thing is, when we do that, we do the full row each >>>> time. In other word, for each query we deserialize 70k+ columns >>>> even if to return only one. I'm willing to bet this is what is killing >>>> your response time. If you want to cache wide rows, I really >>>> suggest you're using the ConcurrentLinkedHashCacheProvider >>>> instead. >>> >>> What happens when using ConcurrentLinkedHashCache? What is the >>> implementation like and why is it better? >> >> With ConcurrentLinkedHashCache, the cache is in the heap. So there >> is no deserialization/copy during gets, so having wide rows is not a >> problem. Outside of the fact that if you're enabling cache on a column >> family with wide rows, you have to keep in mind that we always keep >> full rows in cache. >> > > Wouldn't it move the problem to GC pauses from not being able to clean > up old generation? I am using these rows in concurrenthashmap will get > migrated to old gen.
Kinda, yes, that's why we have a serializing cache :) I mean, caching rows of 70k+ columns is *not* the typical case we've optimized for (https://issues.apache.org/jira/browse/CASSANDRA-1956 should improve here) and so yes neither the serializing cache nor the linked hash one will be perfect in that case. But the serializing cache is just worst in that specific case. -- Sylvain >>> >>>> >>>> I'll also note that this explain the ParNew times too. Deserializing >>>> all those columns from off-heap creates lots of short-lived object, >>>> and since you deserialize 70k+ on each query, that's quite some >>>> pressure on the new gen. Note that the serializing cache is >>>> actually minimizing the use of old gen, because that is the one >>>> that is the one that can create huge GC pauses with big heap, >>>> but it actually put more pressure on the new gen. This is by >>>> design and because new gen is much less of a problem than >>>> old gen. >>> >>> In this scenario would it help if Young generation space is increased? >> >> That's a hard one to answer because GC tuning is a bit of a black >> art, when testing and benchmarking is often key. Having a bigger >> young generation means having young collection kicked less often >> but on the other side it reduces the size for the old generation. >> But again, I don't think the problem is really the GC here, at least not >> primarily. >> >> -- >> Sylvain >> >>> >>>> >>>> -- >>>> Sylvain >>>> >>> >> >
