Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict, So I had a look at the code, and as you say it looked pretty easy to recycle on heap slabs… there is already RACE_ALLOCATED which keeps a strongly referenced pool, however I was thinking in this case of just WeakReferences. In terms of on heap slabs, it seemed to me that recycling

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread Benedict Elliott Smith
Hi Graham, Unfortunately the problem is more difficult than you might think. References to the buffers can persist in-flight to clients long after the memtable is discarded, so you would be introducing a subtle corruption risk for data returned to clients. Unfortunately the current implementation

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict, Ah, my mistake, I had assumed that since the memory backing the off heap ByteBuffer Regions was freed on discard, that the Regions would be immediately recyclable in the on heap case. I guess I’ll just configure for one of the off-heap variants when 2.1 comes out… Any idea of the

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread Benedict Elliott Smith
The current implementation is slower for a single memtable-only read (dependent on workload, but similar ball park to on heap), but can store substantially more per memtable under offheap_objects (again dependent on workload, but as much as 8x in extreme cases) which means more queries are answerab

Re: CMS GC / fragmentation / memtables etc

2014-06-15 Thread graham sanderson
Hi Benedict thanks once again for your explanations (hopefully these are my last questions!)… I just read https://issues.apache.org/jira/browse/CASSANDRA-6694 really quickly, so didn’t follow all the discussions…. one thing to note that was argued in the middle was about the cost of virtual met