[
https://issues.apache.org/jira/browse/LUCENE-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-6169:
----------------------------------
Description:
I open this issue to keep track of the communication with Oracle and OpenJDK
about this:
Basically, what happens: In LUCENE-5588 we added support to FSDirectory to be
able to sync on directory metadata changes (means the contents of the directory
itsself). This is very important on Unix system (maybe also on Windows),
because fsyncing a single file does not necessarily writes the directory's
contents to disk. Lucene uses this for commits. We first do an atomic rename of
the segments file (to make the commit public), but we have to be sure that the
rename operation is written to disk. Because of that we must fsync the
directory.
To enforce this with plain system calls (libc), you open a directory for read
and then call fsync. In java this can be done by opening a FileChannel on the
direczory(for read) and call fc.force() on it.
Unfortunately the commit
http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/e5b66323ae45 in OpenJDK 9 break
this. The corresponding issue is
https://bugs.openjdk.java.net/browse/JDK-8066915. The JDK now explicitly checks
if a file is a directory and disallows opening a FileChannel on it. This breaks
our commit safety.
Because this behaviour is undocumented (not even POSIX has explicit semantics
for syncing directories), we know that it worked at least on MacOSX and Linux.
The code in IOUtils is currently written in a way that it tries to sync the
diretory, but swallows any Exception. So this change does not break Liucene,
but it breaks our commit safety. During testing we assert that the fsync
actually works on Linux and MacOSX, in production code the user will notice
nothing.
We should take action and contact Alan Bateman about his commit and this issue
on the mailing list, possibly through Rory O'Donnell.
was:
I open this issue to keep track of the communication with Oracle and OpenJDK
about this:
Basically, what happens: In LUCENE-5588 we added support to FSDirectory to be
able to sync on directory metadata changes (means the contents of the directory
itsself). This is very important on Unix system (maybe also on Windows),
because fsyncing a single file does not necessarily writes the directory's
contents to disk. Lucene uses this for commits. We first do an atomic rename of
the segments file (to make the commit public), but we have to be sure that the
rename operation is written to disk. Because of that we must fsync the
directory.
To enforce this, you open a directory for read and then call fsync. In java
this can be done by opening a FileChannel on the direczory(for read) and call
fc.force() on it.
Unfortunately the commit
http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/e5b66323ae45 in OpenJDK 9 break
this. The corresponding issue is
https://bugs.openjdk.java.net/browse/JDK-8066915. The JDK now explicitly checks
if a file is a directory and disallows opening a FileChannel on it. This breaks
our commit safety.
Because this behaviour is undocumented (not even POSIX has explicit semantics
for syncing directories), we know that it worked at least on MacOSX and Linux.
The code in IOUtils is currently written in a way that it tries to sync the
diretory, but swallows any Exception. So this change does not break Liucene,
but it breaks our commit safety. During testing we assert that the fsync
actually works on Linux and MacOSX, in production code the user will notice
nothing.
We should take action and contact Alan Bateman about his commit and this issue
on the mailing list, possibly through Rory O'Donnell.
> Recent Java 9 commit breaks fsync on directory
> ----------------------------------------------
>
> Key: LUCENE-6169
> URL: https://issues.apache.org/jira/browse/LUCENE-6169
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Reporter: Uwe Schindler
> Labels: Java9
>
> I open this issue to keep track of the communication with Oracle and OpenJDK
> about this:
> Basically, what happens: In LUCENE-5588 we added support to FSDirectory to be
> able to sync on directory metadata changes (means the contents of the
> directory itsself). This is very important on Unix system (maybe also on
> Windows), because fsyncing a single file does not necessarily writes the
> directory's contents to disk. Lucene uses this for commits. We first do an
> atomic rename of the segments file (to make the commit public), but we have
> to be sure that the rename operation is written to disk. Because of that we
> must fsync the directory.
> To enforce this with plain system calls (libc), you open a directory for read
> and then call fsync. In java this can be done by opening a FileChannel on the
> direczory(for read) and call fc.force() on it.
> Unfortunately the commit
> http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/e5b66323ae45 in OpenJDK 9 break
> this. The corresponding issue is
> https://bugs.openjdk.java.net/browse/JDK-8066915. The JDK now explicitly
> checks if a file is a directory and disallows opening a FileChannel on it.
> This breaks our commit safety.
> Because this behaviour is undocumented (not even POSIX has explicit semantics
> for syncing directories), we know that it worked at least on MacOSX and
> Linux. The code in IOUtils is currently written in a way that it tries to
> sync the diretory, but swallows any Exception. So this change does not break
> Liucene, but it breaks our commit safety. During testing we assert that the
> fsync actually works on Linux and MacOSX, in production code the user will
> notice nothing.
> We should take action and contact Alan Bateman about his commit and this
> issue on the mailing list, possibly through Rory O'Donnell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
