Hi all.
I seem to have a situation where ReaderManager is reducing a refCount
to 0 before it actually releases all its references.
It's difficult because it's all mixed up in our framework for multiple
ReaderManagers, which I'm still not convinced works because the
concurrency is impossible to figure out, and probably won't be allowed
to publish in order to have anyone at Lucene look at it either. (Which
is why I hope that someone at Lucene figures out how to manage more
than one index reliably one day...)
The stack trace trying to close the directory is just trying to
refresh the reader, but I guess this reader was the last one using a
Directory, so now we're closing that as well:
java.lang.RuntimeException: Resources inside the directory did not
get closed before closing the directory
at
com.acme.storage.textindex.store.CloseCheckingDirectory.close(CloseCheckingDirectory.java:109)
at
com.acme.storage.textindex.index.DefaultIndexReaderSharer$IndexReaderWrapper.release(DefaultIndexReaderSharer.java:146)
at
com.acme.storage.textindex.index.DefaultIndexReaderSharer$IndexReaderWrapper.access$100(DefaultIndexReaderSharer.java:77)
at
com.acme.storage.textindex.index.DefaultIndexReaderSharer.release(DefaultIndexReaderSharer.java:45)
at
com.acme.storage.textindex.DefaultTextIndex$WrappingReaderManager$1.doClose(DefaultTextIndex.java:370)
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
at
com.acme.storage.textindex.DefaultTextIndex$WrappingReaderManager.decRef(DefaultTextIndex.java:331)
at
com.acme.storage.textindex.DefaultTextIndex$WrappingReaderManager.decRef(DefaultTextIndex.java:306)
at
org.apache.lucene.search.ReferenceManager.release(ReferenceManager.java:274)
at
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:189)
at
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
The stack trace which opened the resource and didn't close it is
apparently the first reader which ReaderManager:
Caused by: java.lang.RuntimeException: unclosed IndexInput: _7d.tvd
at
com.acme.storage.textindex.store.CloseCheckingDirectory.addOpenResource(CloseCheckingDirectory.java:82)
at
com.acme.storage.textindex.store.CloseCheckingDirectory.openInput(CloseCheckingDirectory.java:57)
at
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:144)
at
org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at
com.acme.storage.textindex.index.DefaultIndexReaderSharer$CustomReaderManager.<init>(DefaultIndexReaderSharer.java:164)
But if it's the first reader held by the ReaderManager, I wouldn't
expect the refCount to be 0, so it shouldn't be closing the directory.
I can't reproduce this myself, so I can't just dump out conveniently
placed messages to figure out how it's happening...
But has anyone else seen something like this?
CustomReaderManager is probably shareable, it just does this:
private static class CustomReaderManager extends
ReferenceManager<DirectoryReader> {
private CustomReaderManager(Directory directory) throws IOException {
current =
UnInvertingDirectoryReader.wrap(DirectoryReader.open(directory));
}
@Override
protected void decRef(DirectoryReader reference) throws IOException {
reference.decRef();
}
@Override
protected DirectoryReader refreshIfNeeded(DirectoryReader
referenceToRefresh) throws IOException {
return DirectoryReader.openIfChanged(referenceToRefresh);
}
@Override
protected boolean tryIncRef(DirectoryReader reference) {
return reference.tryIncRef();
}
@Override
protected int getRefCount(DirectoryReader reference) {
return reference.getRefCount();
}
}
So basically the same as the normal one, except that it wraps the
reader in an UnInvertingDirectoryReader. The only reason we're forced
to subclass the manager to do this is that if we don't, each
UnInvertingDirectoryReader becomes a new instance, and basic caching
stuff stops working in some way.
DefaultIndexReaderSharer#release() is like this:
private synchronized void release(DirectoryReader reader) {
try {
if (readerManager == null) {
reader.decRef(); // assumes we don't own it anymore
return;
}
readerManager.release(reader);
if (reader.getRefCount() == 1) {
readerManager.close(); // 👈 close
happens from here
readerManager = null;
reader.directory().close();
}
} catch (IOException e) {
Logger.getLogger(getClass()).warn("Error closing reader", e);
}
}
So it's releasing the reader and then checking the ref count. If the
count is now 1, it closes it, but who knows, maybe this can race,
despite the synchronized keyword put here explicitly to prevent that,
based on the advice last time.
Basically, it is still essentially impossible to manage indexes
without getting weird errors like this. I can't even tell whether
we're doing something wrong or whether Lucene is simply not closing
the files it's opening. :/
TX
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
