Thanks Aaron, I found the implementation in CLibrary.trySkipCache() method which uses fadvise DONTNEED flag after going through https://issues.apache.org/jira/browse/CASSANDRA-1470
I also came across the link mentioned in JIRA http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html?showComment=1303235497682#c2572106601600642254 which says 2.6.29 version above has implemented madvise SEQUENTIAL in a better manner. So for memory mapped files, compaction can do a madvise SEQUENTIAL instead of current DONTNEED flag after detecting appropriate OS versions. Will this help? -- Ravi On Thu, Dec 6, 2012 at 8:19 AM, aaron morton <aa...@thelastpickle.com>wrote: > Background http://en.wikipedia.org/wiki/Memory-mapped_file > > Is it going to load only relevant pages per SSTable on read or is it going > to load an entire SSTable on first access? > > It will load what is requested, and maybe some additional data taking into > account the amount of memory available for caches. > > Say suppose compaction kicks in. Will it then evict hot MMapped pages for > read and substitute it with a lot of pages involving full SSTables? > > Some file access in cassandra, such as compaction, hints to the OS that > the reads should not be cached. Technically is uses posix_fadvise if you > want to look it up. > > Cheers > > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 5/12/2012, at 11:04 PM, Ravikumar Govindarajan < > ravikumar.govindara...@gmail.com> wrote: > > Thanks Aaron, > > I am not quite clear on how MMap loads SSTables other than the fact that > it kicks in only during a first-time access > > Is it going to load only relevant pages per SSTable on read or is it going > to load an entire SSTable on first access? > > Say suppose compaction kicks in. Will it then evict hot MMapped pages for > read and substitute it with a lot of pages involving full SSTables? > > -- > Ravi > > On Wed, Dec 5, 2012 at 1:22 AM, aaron morton <aa...@thelastpickle.com>wrote: > >> Will MMapping data files be detrimental for reads, in this case? >> >> No. >> >> In general, when should we opt for MMap data files and what are the >> factors that need special attention when enabling the same? >> >> mmapping is the default, so I would say use it until you have a reason >> not to. >> >> mmapping will map the entire file, but pages of data are read into memory >> on demand and purged when space is needed. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 4/12/2012, at 11:59 PM, Ravikumar Govindarajan < >> ravikumar.govindara...@gmail.com> wrote: >> >> Our current SSTable sizes are far greater than RAM. {150 Gigs of data, >> 32GB RAM}. Currently we run with mlockall and mmap_index_only options and >> don't experience swapping at all. >> >> We use wide rows and size-tiered-compaction, so a given key will >> definitely be spread across multiple sstables. Will MMapping data files be >> detrimental for reads, in this case? >> >> In general, when should we opt for MMap data files and what are the >> factors that need special attention when enabling the same? >> >> -- >> Ravi >> >> >> > >
