[
https://issues.apache.org/jira/browse/SOLR-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717028#comment-15717028
]
David Smiley commented on SOLR-9824:
------------------------------------
Work-around: Thanks to some additional conditions of when the lastDocInBatch
condition happens, it's possible to work-around this performance bug by setting
{{-Dsolr.cloud.replication.runners=2}} (not 1). I don't really like it at 2
but since it fixes the bug, I'm going with it. In addition, in my environment
I've set {{-Dsolr.cloud.replication.poll-queue-time-ms=1000}}; the default is
25.
solr.cloud.replication.poll-queue-time-ms: The fact that this defaults to a
measly 25ms is too low IMO. I think it should be at least 250 -- which happens
to be the default in ConcurrentUpdateSolrClient. The lower it is, the greater
likelihood of more indexing overhead including log messages. The greater it
is, it could delay a /update connection from completing up to this amount of
time.
AddUpdateCommand.pollQueueTime defaults to 0, which is only modified to
{{-Dsolr.cloud.replication.poll-queue-time-ms=25}} by javabin. So if you send
data to Solr in anything other than javabin, boy are you in for some HTTP
connection frenzy (I've tried). I insist we set this to a reasonable number.
Perhaps 250 as per my other suggestion, and overridable using the same system
property.
> Documents indexed in bulk are replicated using too many HTTP requests
> ---------------------------------------------------------------------
>
> Key: SOLR-9824
> URL: https://issues.apache.org/jira/browse/SOLR-9824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 6.3
> Reporter: David Smiley
>
> This takes awhile to explain; bear with me. While working on bulk indexing
> small documents, I looked at the logs of my SolrCloud nodes. I noticed that
> shards would see an /update log message every ~6ms which is *way* too much.
> These are requests from one shard (that isn't a leader/replica for these docs
> but the recipient from my client) to the target shard leader (no additional
> replicas). One might ask why I'm not sending docs to the right shard in the
> first place; I have a reason but it's besides the point -- there's a real
> Solr perf problem here and this probably applies equally to
> replicationFactor>1 situations too. I could turn off the logs but that would
> hide useful stuff, and it's disconcerting to me that so many short-lived HTTP
> requests are happening, somehow at the bequest of DistributedUpdateProcessor.
> After lots of analysis and debugging and hair pulling, I finally figured it
> out.
> In SOLR-7333 ([~tpot]) introduced an optimization called
> {{UpdateRequest.isLastDocInBatch()}} in which ConcurrentUpdateSolrClient will
> poll with a '0' timeout to the internal queue, so that it can close the
> connection without it hanging around any longer than needed. This part makes
> sense to me. Currently the only spot that has the smarts to set this flag is
> {{JavaBinUpdateRequestCodec.unmarshal.readOuterMostDocIterator()}} at the
> last document. So if a shard received docs in a javabin stream (but not
> other formats) one would expect the _last_ document to have this flag.
> There's even a test. Docs without this flag get the default poll time; for
> javabin it's 25ms. Okay.
> I _suspect_ that if someone used CloudSolrClient or HttpSolrClient to send
> javabin data in a batch, the intended efficiencies of SOLR-7333 would apply.
> I didn't try. In my case, I'm using ConcurrentUpdateSolrClient (and BTW
> DistributedUpdateProcessor uses CUSC too). CUSC uses the RequestWriter
> (defaulting to javabin) to send each document separately without any leading
> marker or trailing marker. For the XML format by comparison, there is a
> leading and trailing marker (<stream> ... </stream>). Since there's no outer
> container for the javabin unmarshalling to detect the last document, it marks
> _every_ document as {{req.lastDocInBatch()}}! Ouch!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
