Kevin Liang created SOLR-18476:
----------------------------------
Summary: Support file suffix based MMapDirectory preload
Key: SOLR-18476
URL: https://issues.apache.org/jira/browse/SOLR-18476
Project: Solr
Issue Type: Improvement
Reporter: Kevin Liang
*Context:*
Solr uses various DirectoryFactory implementations to interface with the raw
index on disk. When Solr requests index files, it often is actually going to
MMap'ed off-heap memory rather than disk itself (disk reads being slow). By
default, Solr starts up "cold" however there is existing support for
preload=true as a configuration param of MMapDirectoryFactory which will
attempt to load as much of the index directory into off-heap memory page cache
as possible. This preload is done both at startup time, but also after merges,
and would avoid issues such as first queries after merge being much slower.
The issue with this is that when our index is large, some files should be given
priority rather than the indiscriminate preload=true. An example being that
dense vector search is highly dependent on having the graph and vectors
in-memory for traversal and scoring (.vex, .vem, .vec). Lucene provides this
ability in their API, however Solr is not currently leveraging it
*Proposed Change:*
Add a <str name="preloadExtensions">...</str> property to MMapDirectoryFactory
(or if we want to be more cautious, could consider implementing a new
"SelectiveMMapDirectoryFactory"). The provided ordered list of file extensions
would be attempted to be preloaded into page cache in that order and take
precedent over other index files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
