[
https://issues.apache.org/jira/browse/SOLR-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Liang updated SOLR-18476:
-------------------------------
Description:
*Context:*
Solr uses various DirectoryFactory implementations to interface with the raw
index on disk. When Solr requests index files, it often is actually going to
MMap'ed off-heap memory rather than disk itself (disk reads being slow). By
default, Solr starts up "cold" however there is existing support for
preload=true as a configuration param of MMapDirectoryFactory which will
attempt to load as much of the index directory into off-heap memory page cache
as possible. This preload is done both at startup time, but also after merges,
and would avoid issues such as first queries after merge being much slower.
The issue with this is that when our index is large, some files should be given
priority rather than the indiscriminate preload=true. An example being that
dense vector search is highly dependent on having the graph and vectors
in-memory for traversal and scoring (.vex, .vem, .vec). The inverse can also be
true that we may not want to populate the page cache with stored fields, as
would be done in an all-or-nothing preload. Lucene provides this ability to
choose files in their API, however Solr is not currently leveraging it
*Proposed Change:*
Add a <str name="preloadExtensions">...</str> property to MMapDirectoryFactory
(or if we want to be more cautious, could consider implementing a new
"SelectiveMMapDirectoryFactory"). The provided ordered list of file extensions
would be attempted to be preloaded into page cache in that order and take
precedent over other index files.
was:
*Context:*
Solr uses various DirectoryFactory implementations to interface with the raw
index on disk. When Solr requests index files, it often is actually going to
MMap'ed off-heap memory rather than disk itself (disk reads being slow). By
default, Solr starts up "cold" however there is existing support for
preload=true as a configuration param of MMapDirectoryFactory which will
attempt to load as much of the index directory into off-heap memory page cache
as possible. This preload is done both at startup time, but also after merges,
and would avoid issues such as first queries after merge being much slower.
The issue with this is that when our index is large, some files should be given
priority rather than the indiscriminate preload=true. An example being that
dense vector search is highly dependent on having the graph and vectors
in-memory for traversal and scoring (.vex, .vem, .vec). The inverse can also be
true that we may not want to populate the page cache with postings or stored
fields, as would be done in an all-or-nothing preload. Lucene provides this
ability to choose files in their API, however Solr is not currently leveraging
it
*Proposed Change:*
Add a <str name="preloadExtensions">...</str> property to MMapDirectoryFactory
(or if we want to be more cautious, could consider implementing a new
"SelectiveMMapDirectoryFactory"). The provided ordered list of file extensions
would be attempted to be preloaded into page cache in that order and take
precedent over other index files.
> Support file suffix based MMapDirectory preload
> -----------------------------------------------
>
> Key: SOLR-18476
> URL: https://issues.apache.org/jira/browse/SOLR-18476
> Project: Solr
> Issue Type: Improvement
> Reporter: Kevin Liang
> Priority: Major
>
> *Context:*
> Solr uses various DirectoryFactory implementations to interface with the raw
> index on disk. When Solr requests index files, it often is actually going to
> MMap'ed off-heap memory rather than disk itself (disk reads being slow). By
> default, Solr starts up "cold" however there is existing support for
> preload=true as a configuration param of MMapDirectoryFactory which will
> attempt to load as much of the index directory into off-heap memory page
> cache as possible. This preload is done both at startup time, but also after
> merges, and would avoid issues such as first queries after merge being much
> slower.
> The issue with this is that when our index is large, some files should be
> given priority rather than the indiscriminate preload=true. An example being
> that dense vector search is highly dependent on having the graph and vectors
> in-memory for traversal and scoring (.vex, .vem, .vec). The inverse can also
> be true that we may not want to populate the page cache with stored fields,
> as would be done in an all-or-nothing preload. Lucene provides this ability
> to choose files in their API, however Solr is not currently leveraging it
> *Proposed Change:*
> Add a <str name="preloadExtensions">...</str> property to
> MMapDirectoryFactory (or if we want to be more cautious, could consider
> implementing a new "SelectiveMMapDirectoryFactory"). The provided ordered
> list of file extensions would be attempted to be preloaded into page cache in
> that order and take precedent over other index files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
