[
https://issues.apache.org/jira/browse/SOLR-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-7333:
---------------------------------
Attachment: SOLR-7333.patch
Here's a first pass at a patch that uses a 25ms pollQueueTime when processing a
batch of documents. The main idea here is that the javabin unmarshalling code
can detect when it sees the last doc in a batch and can pass that hint down the
line, eventually to the ConcurrentUpdateSolrServer the leader uses to stream
docs to replicas. CUSS uses that hint to poll the queue for 0 (vs. waiting the
25ms). This helps stream more docs from the leader to replica per request,
which keeps the requests processed by leaders and replicas nearly the same and
reduces round-trips per batch between leader and replica. The hint about being
the last doc in a batch is necessary to avoid the wait when processing docs
one-by-one or when the last doc in a batch has been processed, i.e. poll the
queue with a brief wait if more docs are available but don't wait if not.
Currently, the pollQueueTime is hardcoded to 25 ms, but I suppose we could make
that configurable. The key is to use a short wait so I felt 25 ms should be
sufficient for most indexing applications.
Lastly, I added the {{isLastDocInBatch}} flag as a member to UpdateRequest
instead of including it into the params because CUSS checks for params changing
while processing UpdateRequests in a batch and treats a change in parameters as
a separate request, which is what we're trying to avoid here.
> Make the poll queue time configurable and use knowledge that a batch is being
> processed to poll efficiently
> -----------------------------------------------------------------------------------------------------------
>
> Key: SOLR-7333
> URL: https://issues.apache.org/jira/browse/SOLR-7333
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Timothy Potter
> Assignee: Timothy Potter
> Attachments: SOLR-7333.patch
>
>
> {{StreamingSolrClients}} uses {{ConcurrentUpdateSolrServer}} to stream
> documents from leader to replica, by default it sets the {{pollQueueTime}}
> for CUSS to 0 so that we don't impose an unnecessary wait when processing
> single document updates or the last doc in a batch. However, the downside is
> that replicas receive many more update requests than leaders; I've seen up to
> 40x number of update requests between replica and leader.
> If we're processing a batch of docs, then ideally the poll queue time should
> be greater than 0 up until the last doc is pulled off the queue. If we're
> processing a single doc, then the poll queue time should always be 0 as we
> don't want the thread to wait unnecessarily for another doc that won't come.
> Rather than force indexing applications to provide this optional parameter in
> an update request, it would be better for server-side code that can detect
> whether an update request is a single document or batch of documents to
> override this value internally, i.e. it'll be 0 by default, but since
> {{JavaBinUpdateRequestCodec}} can determine when it's seen the last doc in a
> batch, it can override the pollQueueTime to something greater than 0.
> This means that current indexing clients will see a boost when doing batch
> updates without making any changes on their side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
