[
https://issues.apache.org/jira/browse/LUCENE-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-3237:
---------------------------------------
Attachment: LUCENE-3237.patch
While it's hard to imagine an OS that would (could?) separately track
which (of possibly many) file handles had made changes to the dirty
buffers in the OS's IO cache for a given file, such that closing the
open (for write) file, opening it again later, fsync'ing that, would
actually fail to move some bytes to durable storage, it has always
made me a bit nervous that Lucene closes a file and later reopens it
to fsync. This may simply be paranoia :)
Still, I tried a prototype patch:
* Removed Directory.sync(Collection<String>) and added
IndexOutput.sync. Ie, you now must sync before closing the
IndexOutput.
* Fixed IndexWriter to prevent .close() of all IndexOutputs (it
wraps the incoming Directory to do this), until 1) the output is
sync'd, or 2) the file is deleted.
* During commit, IndexWriter retrieves the still-held-open
IndexOutputs and asks them to sync.
This also has the nice side effect of removing the sneaky stale file
tracking that FSDirectory was doing (we had wanted to remove this in
LUCENE-5570).
One downside of this approach is it means IndexWriter holds file
descriptors open, one per written but not sync'd/deleted file, so it
sort of acts like an IndexReader in this regard. Though, the count
will be lower, since it's only newly written files since open/last
commit.
Another big problem with this approach is: if you open near-real-time
readers, this now means we are opening IndexInputs when there are
still IndexOutputs open against those files, which will probably make
some Directory impls unhappy. (I had to turn off this check in MDW).
The patch is not well tested but I did get it to the point where
TestDemo passes...
There are serious nocommits too, e.g. NRTCachingDir is now broken, and
MDW now longer crashes on close.
> FSDirectory.fsync() may not work properly
> -----------------------------------------
>
> Key: LUCENE-3237
> URL: https://issues.apache.org/jira/browse/LUCENE-3237
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Reporter: Shai Erera
> Attachments: LUCENE-3237.patch
>
>
> Spinoff from LUCENE-3230. FSDirectory.fsync() opens a new RAF, sync() its
> FileDescriptor and closes RAF. It is not clear that this syncs whatever was
> written to the file by other FileDescriptors. It would be better if we do
> this operation on the actual RAF/FileOS which wrote the data. We can add
> sync() to IndexOutput and FSIndexOutput will do that.
> Directory-wise, we should stop syncing on file names, and instead sync on the
> IOs that performed the write operations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
