[
https://issues.apache.org/jira/browse/LUCENE-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015940#comment-16015940
]
David Smiley commented on LUCENE-7834:
--------------------------------------
Here's a use-case: Lots of searchable fields, but most of them are copied to a
catch-all search field. Most searches don't reference the source fields (i.e.
most searches aren't specific about exactly what field needs to match), but it
needs to be supported for the small number of queries that do. By not loading
them eagerly, it saves memory and is more NRT friendly.
Any thoughts on adding IndexReaderWarmer support to IndexSearcher? I think
many/most apps warm their index already, and perhaps most that don't arguably
should.
A half baked idea I was just exploring was a PostingsFormat wrapper that
lazy-loads. But a problem there is knowing which fields exist without using
the underlying real PostingsFormat (which would trigger memory & stuff we are
trying to avoid). And of course you can't then tell the underlying postings
format which field exactly you want alone (and not the others) so this seems to
be totally unworkable. It'd be nice if it did though; then you could make
*any* PostingsFormat lazy load... you could even have a wrapping PostingsFormat
that could unload if it so chose. Wishful thinking, alas. Maybe there is a
possibility there still by concocting a special combo of CFS and
PerFieldPostingsFormat.
> BlockTree's terms index should be loaded into memory lazily
> -----------------------------------------------------------
>
> Key: LUCENE-7834
> URL: https://issues.apache.org/jira/browse/LUCENE-7834
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Reporter: David Smiley
> Assignee: David Smiley
> Attachments: LUCENE_7834_BlockTreeLazyFST.patch
>
>
> I propose that BlockTree delay loading the FST prefix terms index into memory
> until {{terms(String)}} is first called. This seems like how it should work
> since if someone wants to eager load then they can use {{IndexReaderWarmer}}.
> By making the FST lazy load, we can be more NRT friendly (if some fields are
> rarely used), and also save memory (if some fields are rarely used).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
