Sure, if your analysis is correct we can either add the comment to cassandray.yaml or NEWS.txt, or both.
Could you leave a comment with your analysis and suggestion to CASSANDRA-11993? When Branimir is back from holiday he can probably confirm easily if that's the case. On Wed, Jun 22, 2016 at 5:44 PM, Nimi Wariboko Jr <n...@channelmeter.com> wrote: > Hi Stefania, > > Thanks for the explanation - I think it may be more likely I'm hitting > MEMORY_USAGE_THRESHOLD > (file_cache_size_in_mb, which is 512mb by default), as the table itself is > very large (a couple TB over 12 nodes). In that case, due to the changes > outlined CASSANDRA-9548/CASSANDRA-8709/CASSANDRA-5863, there should > probably be a warning in 3.6 that buffer_pool_use_heap_if_exhausted should > be set to false if you still have some tables on SnappyCompression. > > Nimi > > On Wed, Jun 22, 2016 at 12:14 AM, Stefania Alborghetti < > stefania.alborghe...@datastax.com> wrote: > > > Hi Nimi > > > > I am not familiar with how CASSANDRA-5863 uses the buffer pool yet, but I > > do know how the buffer pool works > > > > It has a local pool, which is thread local, and takes chunks from a > global > > pool. These are allocated off heap and aligned to the page cache > > boundaries, see here > > < > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L276 > > >. > > > > > > The buffer pool will allocate according to > > buffer_pool_use_heap_if_exhausted > > only in two cases: > > > > - The global pool has run out of chunks, which happens when there is > an > > OOM exception or when MEMORY_USAGE_THRESHOLD is exhausted, see here > > < > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L259 > > > > > . > > - The requested buffer size is larger than a chunk, currently > > CHUNK_SIZE, 64 kb. > > > > To understand this behaviour look at maybeTakeFromPool > > < > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L124 > > >(), > > the line that takes from the local pool is the last one and follow from > > there. > > > > The original ticket for the buffer pool is CASSANDRA-8897 > > <https://issues.apache.org/jira/browse/CASSANDRA-8897>if you need more > > details, and there is CASSANDRA-9468 > > <https://issues.apache.org/jira/browse/CASSANDRA-9468> to extend it to > > allow chunks with bigger or smaller sizes (point 4). > > > > I've left a comment on CASSANDRA-11993. > > > > I hope this helps, > > Stefania > > > > > > On Wed, Jun 22, 2016 at 4:54 AM, Nimi Wariboko Jr <n...@channelmeter.com > > > > wrote: > > > > > Hi, > > > > > > I'm trying to debug an issue I'm having with Cassandra 3.6+ and Snappy > > > compressed tables - CASSANDRA-11993 (that may be related to > > > CASSANDRA-5863). > > > > > > I'm probably 3% familiar with Cassandra internals, so apologies if I'm > > > wrong here - I was looking at a change in CASSANDRA-5863 where a cache > > was > > > added to the SSTableIterator code path, which may have > > > included org.apache.cassandra.utils.memory.BufferPool into the read > path. > > > > > > My confusion comes from here - AFAICT, when the BufferPool needs to > > > allocate more memory, it calls allocate(int size, boolean onHeap) ( > > > > > > > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L82 > > > ). > > > The value for whether onHeap is true is > > > constant ALLOCATE_ON_HEAP_WHEN_EXAHUSTED. > > > > > > ALLOCATE_ON_HEAP_WHEN_EXAHUSTED is defined by the yaml > > > setting buffer_pool_use_heap_if_exhausted which is true by default (and > > was > > > so in my install). > > > > > > Further still ( > > > > > > > > > https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java > > > ), > > > I can't see any logic where BufferPool will actually allocate off heap > > > memory - it seems, due to the buffer_pool_use_heap_if_exhausted > setting, > > > always allocate memory on heap. It never checks if the off heap memory > is > > > exhausted (AFAICT from these 3 files). > > > > > > Sure enough, if I set buffer_pool_use_heap_if_exhausted, my issue at > > > CASSANDRA-11993 goes away (I think? I'm having a number of issues > lately > > > and I may have traded one exception for another). > > > > > > Is this the correct behavior of buffer_pool_use_heap_if_exhausted? > > > > > > Nimi > > > > > > > > > > > -- > > > > > > [image: datastax_logo.png] <http://www.datastax.com/> > > > > Stefania Alborghetti > > > > Apache Cassandra Software Engineer > > > > |+852 6114 9265| stefania.alborghe...@datastax.com > > > > > > [image: cassandrasummit.org/Email_Signature] > > <http://cassandrasummit.org/Email_Signature> > > > -- Stefania Alborghetti Apache Cassandra Committer |+852 6114 9265| stefania.alborghe...@datastax.com [image: cassandrasummit.org/Email_Signature] <http://cassandrasummit.org/Email_Signature>
