Hi there,
we recently updated our application from lucene 3.0 to 3.6 with the
effect
that (albeit using the SearchManager functionality as described on
http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html)
calls to searcherManager.maybeRefresh() were incredibly slow. e.g.
taking
about 30 seconds after adding one document to the index with an
index of
about 9000 documents. I assumed that we did something wrong with the
configuration as 30 seconds could not be meant with NRT ;-)
Thus we migrated to the latest 4.6 version and indexing speed was
indeed
very good now (with the searcherManager.maybeRefreshBlocking() call
only
taking milliseconds to complete). But after some wore testing we
discovered
that somehow the indexWriter.updateDocument( term, documentToIndex )
functionality wasn't working anymore as expected - at least
somtetimes. It
looks like either the updateDocument method does not longer reliably
delete
the old document before adding a new one - with the result that
older
documents are beeing returned by searches breaking our application.
Unfortunately I'm not able to reproduce the issues in a simple unit
test but
maybe somebody of the lucene experts knows what we are doing wrong
here. Not
sure if it is of any relevance but we are running on Windows with a
64 bit
JDK 7 thus MMapDirectory is beeing used.
Our Index Writer is configured like this:
IndexWriterConfig conf = new IndexWriterConfig(
Version.LUCENE_46,
new LimitTokenCountAnalyzer( new DefaultAnalyzer(),
Integer.MAX_VALUE ) );
conf.setOpenMode( OpenMode.APPEND );
IndexWriter indexWriter = new IndexWriter( FSDirectory.open(
new
File( directoryPath )), conf );
SearcherManager is configured like this:
searcherManager = new SearcherManager(indexWriter, true,
null);
// The anlyzer that we are using looks like this:
public class DefaultAnalyzer extends Analyzer
{
@Override
protected TokenStreamComponents createComponents(final
String
fieldName,
final Reader reader) {
return new TokenStreamComponents(new
WhitespaceTokenizer(LuceneSearchService.LUCENE_VERSION, reader));
}
}
The update of the index looks like this:
// instead of 42 the unique business identifier is used
Long myUniqueBusinessId = 42l;
BytesRef ref = new BytesRef(NumericUtils.BUF_SIZE_LONG);
NumericUtils.longToPrefixCoded(
myUniqueBusinessId.longValue(), 0,
ref );
Term term = new Term( "MY_UNIQUE_BUSINESS_ID", ref );
// this method may be called multiple times with the same
term and
luceneDocumentToIndex parameter
indexWriter.updateDocument( term, luceneDocumentToIndex);
// After performing a couple of updates we execute
searcherManager.maybeRefreshBlocking();
// For searching we are using the following code
searcher = searcherManager.acquire();
// luceneQuery is the query, filter is some sort of
filtering that
we apply, luceneSort is some sorting query
TopDocs topDocs = searcher.search( luceneQuery, filter,
1000,
luceneSort );
// If we perform a query for MY_UNIQUE_BUSINESS_ID it will return
multiple
results instead of just one - this was neither the case with lucene
3.0 nor
3.6
In order to fix the issue I tried couple of things but to now avail.
It
still happens (not all the time though) that the lucene returns two
documents when querying for MY_UNIQUE_BUSINESS_ID instead of just
one
- setting setMaxBufferedDeleteTerms to 1 in the config
conf.setMaxBufferedDeleteTerms( 1 );
- explicetly deleting instead of just updating
indexWriter.deleteDocuments( term );
- ensuring that the field MY_UNIQUE_BUSINESS_ID is stored in the
index and
not just analysed
- trying to delete the document via indexWriter.tryDeleteDocument()
- calling indexWriter.maybeMerge() after the update
- calling indexWriter.commit() after the update
Sorry for the lenghty post but I wanted to include as much
information as
possible. Let me know if something is missing...
Thanks for helping in advance ;-)
Kai
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
