[
https://issues.apache.org/jira/browse/SOLR-10255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855155#comment-17855155
]
Alexey Serba edited comment on SOLR-10255 at 6/15/24 12:22 AM:
---------------------------------------------------------------
I'm wondering what do you think about simply adding support for
{noformat}
stored="false" docValues="true"
{noformat}
to solr.BinaryField field type?
Lucene {{BinaryDocValuesField}} is the only field type in Lucene that does not
offer block level compression (see
[LUCENE-9843|https://issues.apache.org/jira/browse/LUCENE-9843?focusedCommentId=17321327&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17321327]),
so for many use cases where users prefer speed vs compression (see twitter and
amazon use cases in
[LUCENE-9378|https://issues.apache.org/jira/browse/LUCENE-9378]), this is the
only option really.
I think this performance issue is important even with small to medium size
fields, so we don't have to necessary solve the caching part in the same PR
(i.e. support for docValues can be added to BinaryField first and an option for
disabling caching can be added later/separately).
was (Author: alexey):
I'm wondering what do you think about simply adding support for
{noformat}
stored="false" docValues="true"
{noformat}
to solr.BinaryField field type?
Lucene {{BinaryDocValuesField}} is the only field type in Lucene that does not
offer block level compression (see
[LUCENE-9843|https://issues.apache.org/jira/browse/LUCENE-9843?focusedCommentId=17321328&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17321328]),
so for many use cases where users prefer speed vs compression (see twitter and
amazon use cases in
[LUCENE-9378|https://issues.apache.org/jira/browse/LUCENE-9378]), this is the
only option really.
I think this performance issue is important even with small to medium size
fields, so we don't have to necessary solve the caching part in the same PR
(i.e. support for docValues can be added to BinaryField first and an option for
disabling caching can be added later/separately).
> Large psuedo-stored fields via BinaryDocValuesField
> ---------------------------------------------------
>
> Key: SOLR-10255
> URL: https://issues.apache.org/jira/browse/SOLR-10255
> Project: Solr
> Issue Type: Improvement
> Reporter: David Smiley
> Assignee: David Smiley
> Priority: Major
> Attachments: SOLR-10255.patch, SOLR-10255.patch
>
>
> (sub-issue of SOLR-10117) This is a proposal for a better way for Solr to
> handle "large" text fields. Large docs that are in Lucene StoredFields slow
> requests that don't involve access to such fields. This is fundamental to
> the fact that StoredFields are row-stored. Worse, the Solr documentCache
> will wind up holding onto massive Strings. While the latter could be tackled
> on it's own somehow as it's the most serious issue, nevertheless it seems
> wrong that such large fields are in row-stored storage to begin with. After
> all, relational DBs seemed to have figured this out and put CLOBs/BLOBs in a
> separate place. Here, we do similarly by using, Lucene
> {{BinaryDocValuesField}}. BDVF isn't well known in the DocValues family as
> it's not for typical DocValues purposes like sorting/faceting etc. The
> default DocValuesFormat doesn't compress these but we could write one that
> does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
