[
https://issues.apache.org/jira/browse/SOLR-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Sweeney updated SOLR-16487:
----------------------------------
Description:
Currently the polling for replication is fairly simply, but can lead to polling
too often. As an example if you had the following config for commits:
{code:java}
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>60000</maxTime>
</autoSoftCommit> {code}
The current logic would setup polling to be half of the autoCommit time, so
poll every 7.5 seconds. However since a new searcher isn't opened, there will
only be changes reflected every 60 seconds on the leader. We can make this
logic a bit smarter knowing that the replication handler won't reflect changes
until a new searcher is opened.
was:
Currently there are a few inefficiencies related to pull replicas that can
cause increased resource usage on a Solr cluster, particularly if there are a
large number of cores.
The following are the inefficiencies identified:
# The
[RecoveryStrategy.replicate()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L219]
method makes a call to commit to on the leader. This happens whenever a
replica is reloaded. For PULL replicas in particular this isn't necessary since
we can just pull down whatever the latest data is and rely on other mechanisms
to be consistently committing the leader. (As an aside, it seems like forcing a
commit on the leader might never be necessary, but for this I've limited it to
focusing on PULL replicas).
# In a case where the leader has no data yet (index version is 0), then a
non-leader replica will consistently delete and recreate its core due to this
case in IndexFetcher:
[https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L549.]
This can cause unnecessary CPU usage until the leader has data indexed to it.
# The polling for replication is fairly simply, but can lead to polling too
often. As an example if you had the following config for commits:
{code:java}
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>60000</maxTime>
</autoSoftCommit> {code}
The current logic would setup polling to be half of the autoCommit time, so
poll every 7.5 seconds. However since a new searcher isn't opened, there will
only be changes reflected every 60 seconds on the leader. We can make this
logic a bit smarter knowing that the replication handler won't reflect changes
until a new searcher is opened.
> Improve calculation of poll interval for replicas
> -------------------------------------------------
>
> Key: SOLR-16487
> URL: https://issues.apache.org/jira/browse/SOLR-16487
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 9.0, 8.11.2
> Reporter: Justin Sweeney
> Assignee: Justin Sweeney
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the polling for replication is fairly simply, but can lead to
> polling too often. As an example if you had the following config for commits:
> {code:java}
> <autoCommit>
> <maxTime>15000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>60000</maxTime>
> </autoSoftCommit> {code}
> The current logic would setup polling to be half of the autoCommit time, so
> poll every 7.5 seconds. However since a new searcher isn't opened, there will
> only be changes reflected every 60 seconds on the leader. We can make this
> logic a bit smarter knowing that the replication handler won't reflect
> changes until a new searcher is opened.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
