[
https://issues.apache.org/jira/browse/SOLR-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046347#comment-18046347
]
ASF subversion and git services commented on SOLR-17942:
--------------------------------------------------------
Commit 23fe2ef66794a3842477816add6da09684438a4a in solr's branch
refs/heads/branch_10x from Puneet Ahuja
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=23fe2ef6679 ]
SOLR-17942: Raising configurable RAM per thread hard limit using VarHandle
(#3741)
> Raising the hardcoded limit of lucene parameter ramPerThreadHardLimitMB using
> reflection
> ----------------------------------------------------------------------------------------
>
> Key: SOLR-17942
> URL: https://issues.apache.org/jira/browse/SOLR-17942
> Project: Solr
> Issue Type: Task
> Reporter: Puneet Ahuja
> Priority: Major
> Labels: pull-request-available
> Fix For: 10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The parameter ramPerThreadHardLimitMB cannot be larger than 2GB in Lucene,
> which means a single thread cannot write segments larger than 2GB.
> Refer:
> [https://lucene.apache.org/core/9_9_0/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMPerThreadHardLimitMB(int])
> This issue proposes to make this parameter configurable above the 2GB limit,
> so that each thread can write a bigger segment. I plan to use reflection to
> bypass this hard-coded limit in Lucene.
>
> When indexing high dimensional vector data, each segment has its own HNSW
> graph. So more segments mean more graphs to search per shard and more graph
> rebuild work during merges. With this change, a single indexing thread can
> flush fewer, and larger segments, which is generally more resource-efficient
> for vector-heavy workloads.
> Lucene issue: https://github.com/apache/lucene/issues/15296
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
