[
https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979149#action_12979149
]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}As soon as a DWPT is pulled from production for flushing, it loses all
thread affinity and becomes unavailable until its flush finishes. When a thread
needs a DWPT, it tries to pick the one it last had (affinity) but if that one's
busy, it picks a new one. If none are available but we are below our max DWPT
count, it spins up a new one?{quote}
Right.
{quote}With the proposed approach, all docs added (or in the process of being
added) will make it into the flushed segments once the flush returns; newly
added docs after the flush call started may or not make it. But this is fine? I
mean, if the app has stronger requirements then it should externally
sync?{quote}
Ok. The proposed change is simply the thread calling add doc will flush it's
DWPT if needed, take it offline while doing so, and return it when completed.
I think the risk is a new DWPT likely will have been created during flush,
which'd make the returning DWPT inutile?
{quote}Why would we lose them? Wouldn't that DWPT just go back into rotation
once the flush is done?{quote}
Yes, we just need to change the existing code a bit then.
However I think we may still need the global lock for close, eg, today we're
preventing the user from adding docs during close, after this issue is merged
that behavior would change?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: Realtime Branch
>
> Attachments: LUCENE-2324-SMALL.patch, LUCENE-2324-SMALL.patch,
> LUCENE-2324-SMALL.patch, LUCENE-2324-SMALL.patch, LUCENE-2324-SMALL.patch,
> lucene-2324.patch, lucene-2324.patch, LUCENE-2324.patch, test.out, test.out
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
