[
https://issues.apache.org/jira/browse/SOLR-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471495#comment-16471495
]
Cao Manh Dat commented on SOLR-12338:
-------------------------------------
{quote}
Given that some machines these days have dozens of cores and you might have
many SolrCores recovering, we may want to cap the number of threads at some
number or make it configurable or something.
{quote}
{{replayUpdatesExecutor }] is shared through all the SolrCores, therefore how
many SolrCores are recovering won't affect the max number of threads will be
used. Although, make it configurable is a good idea.
{quote}
Yeah, you need that to ensure FIFO.
{quote}
Yeah, but we do not need that flag for the case of LogReplayer, right? Because
we are calling execute method in single-thread.
{quote}
I think what David is getting at is that you are ensuring that tasks are kicked
off in order, but once they are kicked off, you can't guarantee order. So task1
gets taken off the queue, then task 2 is taken, now task 2 gets executed first
when task 1 has it's thread unluckily scheduled by the OS. At least that's how
I read it. But that is not an issue right? Because you don't run an item from
the queue until the one in front of it is fully run right?
{quote}
OrderedExecutor ensuring that tasks are kicked off in order *for a same id*.
Yeah, task1 get taken off the queue only after it finishes.
{quote}
I like how this gives us some control to throttle, I wonder how efficient it is
as documents keep thundering in though - do we gobble up threads and
connections waiting? That is where it's a bummer it's hard to limit those
resources. What are you going to do though? Those requests have to wait
somewhere or we have to start dropping them - and hopefully with NIO2 it's
somewhat efficient to wait on IO.
{quote}
I think we will throttle the incoming updates properly by doing SOLR-12305.
> Replay buffering tlog in parallel
> ---------------------------------
>
> Key: SOLR-12338
> URL: https://issues.apache.org/jira/browse/SOLR-12338
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Cao Manh Dat
> Assignee: Cao Manh Dat
> Priority: Major
> Attachments: SOLR-12338.patch, SOLR-12338.patch
>
>
> Since updates with different id are independent, therefore it is safe to
> replay them in parallel. This will significantly reduce recovering time of
> replicas in high load indexing environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
