[
https://issues.apache.org/jira/browse/SOLR-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118122#comment-18118122
]
Kevin Liang commented on SOLR-18476:
------------------------------------
https://lucene.apache.org/core/9_7_0/core/org/apache/lucene/store/MMapDirectory.html#setPreload(java.util.function.BiPredicate)
> Support file suffix based MMapDirectory preload
> -----------------------------------------------
>
> Key: SOLR-18476
> URL: https://issues.apache.org/jira/browse/SOLR-18476
> Project: Solr
> Issue Type: Improvement
> Reporter: Kevin Liang
> Priority: Major
>
> *Context:*
> Solr uses various DirectoryFactory implementations to interface with the raw
> index on disk. When Solr requests index files, it often is actually going to
> MMap'ed off-heap memory rather than disk itself (disk reads being slow). By
> default, Solr starts up "cold" however there is existing support for
> preload=true as a configuration param of MMapDirectoryFactory which will
> attempt to load as much of the index directory into off-heap memory page
> cache as possible. This preload is done both at startup time, but also after
> merges, and would avoid issues such as first queries after merge being much
> slower.
> The issue with this is that when our index is large, some files should be
> given priority rather than the indiscriminate preload=true. An example being
> that dense vector search is highly dependent on having the graph and vectors
> in-memory for traversal and scoring (.vex, .vem, .vec). The inverse can also
> be true that we may not want to populate the page cache with postings or
> stored fields, as would be done in an all-or-nothing preload. Lucene provides
> this ability to choose files in their API, however Solr is not currently
> leveraging it
> *Proposed Change:*
> Add a <str name="preloadExtensions">...</str> property to
> MMapDirectoryFactory (or if we want to be more cautious, could consider
> implementing a new "SelectiveMMapDirectoryFactory"). The provided ordered
> list of file extensions would be attempted to be preloaded into page cache in
> that order and take precedent over other index files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
