AFAIK the caches do not expire entries under memory pressure, they just hold the number of entries you specify or are unbound in the case of 100%. The caches is based on the ConcurrentLinkedHashMap (http://code.google.com/p/concurrentlinkedhashmap/) and uses the SecondChance eviction policy.
" to keep all keys in memory on all boxes in the cluster" - Not sure exactly what your are saying, but only the keys and rows stored on the node are kept in the nodes caches.
Is your entire keyset active? If not set a sane starting point (default for key cache is 200,000 http://wiki.apache.org/cassandra/StorageConfiguration ) and see what the cache hit's are like. How many keys do you have? What was your hit rate with 100% key cache?
There is a discussion here about cache and heap sizing
http://www.mail-archive.com/user@cassandra.apache.org/msg04704.html With 16GB on the box, 3GB Heap seems a bit small.
The cache settings are an easy way to shot yourself in foot. The best approach is to start small and increase as needed. Using 100% will mean your shoe budget will soon be cut in half :)
Hope that helps.
Aaron
On 19 Oct, 2010,at 02:02 AM, Jedd Rashbrooke <jedd.rashbro...@imagini.net> wrote:
Greetings,
I would like to check my understanding is accurate on how
KeysCached is understood by Cassandra (0.6.5), and then
get some suggestions on settings / OS FS cache interplay.
First - my initial understanding was that if you set KeysCached
to 100%, Cassandra would do a best effort to keep all keys
in memory on all boxes in the cluster - but it would let some
expire if it was going to run out of memory. *Now* I understand
that it actually just exhausts all its memory and falls over if you
set this value unrealistically high. Is this right?
Assuming this is right ... I'm curious on the optimum settings
for KeysCached. Previously we've had problems here with
boxes doing massive STW GC's, among other things - and
we've been working towards a configuration that is a bit more
stable and predictable, if not as blazingly fast as a brand new
cluster unburdened by much data but with a configuration that
means it'll drive itself into oblivion after several days. ;)
With that in mind, we're trying to keep JVM heap small - about
3GB is our sweet spot so far - after experimenting with numbers
as big as 30GB. Even at 30GB we'd be unable to have 100%
KeysCached anyway - so it was always going to require a
tradeoff - and now we're trying to guess at the best number.
We're using boxes with 16GB, and have an expectation that
the OS will cache a lot of this stuff pretty intelligently - but I
fear that compactions and other activities will mean that the
keys won't always have priority for FS cache. Short of catting
the index files to /dev/null periodically (not a serious suggestion)
has anyone gleaned some insights on the best tradeoffs for
KeysCached where the raw size of your Keys is going to be
at least 10x the size of your available memory? Do you go
small and hope the OS saves you, or do you try to go as big
(but finite) as your JVM will usefully let you and hope
Cassandra caches the best set of keys for your usage profile?
taa,
Jedd.
