[
https://issues.apache.org/jira/browse/LUCENE-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904712#comment-14904712
]
Michael McCandless commented on LUCENE-6813:
--------------------------------------------
bq. I don't fully understand the problem but to me OfflineSorter is thread safe
Sorry I'm still trying to isolate exactly what the issue is ... I'll fixup the
issue title once I have more of a clue.
I think the problem is (maybe) that {{OfflineSorter.sort}} currently removes
its output path well before writing to it, and so if the caller is relying on
{{Files.createTempFile}} to "pick" a unique filename across threads, which BKD
is doing, then this can illegally re-use the same output Path across threads.
But I'm not certain this is the problem, I need to get the thread test online
to see if I can repro/understand outside of BKD's usage.
bq. Also, this looks suspicious to me in OfflineSorter:
If I remove that {{try/catch}} then {{Files.move}} is angry because it cannot
be ATOMIC_MOVE across volumes ... can I just remove the ATOMIC_MOVE option (and
the {{try/catch}})? Why must this be atomic?
> OfflineSorter.sort isn't thread-safe
> ------------------------------------
>
> Key: LUCENE-6813
> URL: https://issues.apache.org/jira/browse/LUCENE-6813
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6813.patch
>
>
> The new BKD tree classes, and NumericRangeTree (just a 1D BKD tree),
> make heavy use of OfflineSorter to build their data structures at
> indexing time when the number of indexed documents is biggish.
> But when I was first building them (LUCENE-6477), I hit a thread
> safety issue in OfflineSorter, and at that time I just worked around
> it by creating my own private temp directory each time I need to write
> a BKD tree.
> This workaround is sort of messy, and it causes problems with "pending
> delete" files on Windows when we try to remove that temp directory,
> causing test failures like
> http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/5149/
> I think instead we should fix the root cause ... i.e. make
> OfflineSorter thread safe. It looks like it's simple...
> Separately I'd like to somehow fix these BKD tests to catch any leaked
> file handles ... I'm not sure they are today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
