I've noticed a few threads on this so far... maybe it's useful or maybe
somebody's already done this, or maybe it's insane and bug-prone.
Anyways, our application requires lucene to act as a non-critical
database, as in each record is composed of denormalized data derived
from the real DBMS. The index can be regenerated at any time from the
database. However, information added to the index must be searchable
immediately after being added. The index is written to concurrently by
many users. Therefore, flushing the IndexWriter to disk, and re-opening
a IndexReader is not really feasible. Therefore, I worked up this hack
to compensate.
Note that this solution precludes multiple readers from reading an
index. Also, a reader cannot be allowed to delete documents (but
really, why can you delete using a reader, anyway? Or has this been
deprecated?)
Essentially, a IndexWriter owns a IndexReader, and to obtain a reader,
you call Indexwriter.getReader(). Whenever the writer is written to, a
new reader is formed, composed of the IndexWriter's SegmentInfos (since
a reader and writer essentially share copies of both of these structures
anyways). It's essentially an in-memory swap rather than reading the
segment infos back from disk after the writer has written them.
I've attached the patch based on the current dev code. Basically it
implements doAfterFlush(), and adds getReader() and addNotifier()
methods. The notifier is simply so that anybody using a Searcher can be
notified that the underlying reader has changed, and the Searcher should
be re-opened.
Something like this:
writer.addNotifier(new WriterUpdateNotifier()
{
public void onUpdate(IndexWriter writer, IndexReader r)
{
// The reader and writer has been updated, rebuild
the searchers
readers[readers.length - 1] = r;
try
{
reader = new MultiReader(readers);
}
catch (IOException e)
{
e.printStackTrace();
}
reopenSearcher();
}
});
This is currently working well in a production system and is working
quite well. It has been load tested, and well, our users are load
testing it for us as well :-). However, see my previous post about the
ArrayIndexOutOfBoundsException, although I don't see how this could be
the cause... but maybe, since nobody else gets the problem. However, I
haven't modified the writer at all, and I am never modifying the index
with the Reader.
So feel free to tell me this is crazy... I'm just throwing it out there.
Thanks
266,285d268
< /** Zigtag added **/
< private IndexReader reader;
<
< private List<WriterUpdateNotifier> notifiers = new ArrayList<WriterUpdateNotifier>();
<
< public IndexReader getReader() throws IOException
< {
< if (reader == null)
< {
< reader = IndexReader.open(directory);
< }
< return reader;
< }
<
< public void addNotifier(WriterUpdateNotifier notifier)
< {
< notifiers.add(notifier);
< }
< /** END Zigtag added **/
<
1857,1914c1840,1842
< // Zigtag added to this class:
< void doAfterFlush()
< throws IOException
< {
< final boolean closeDirectory = false;
< reader = (IndexReader) new SegmentInfos.FindSegmentsFile(directory)
< {
<
< protected Object doBody(String segmentFileName) throws CorruptIndexException, IOException
< {
<
< SegmentInfos infos = segmentInfos;
< //infos.read(directory, segmentFileName);
<
< IndexReader reader;
<
< if (infos.size() == 1)
< { // index is optimized
< reader = SegmentReader.get(infos, infos.info(0), closeDirectory);
< }
< else
< {
<
< // To reduce the chance of hitting FileNotFound
< // (and having to retry), we open segments in
< // reverse because IndexWriter merges & deletes
< // the newest segments first.
<
< IndexReader[] readers = new IndexReader[infos.size()];
< for (int i = infos.size() - 1; i >= 0; i--)
< {
< try
< {
< readers[i] = SegmentReader.get(infos.info(i));
< }
< catch (IOException e)
< {
< // Close all readers we had opened:
< for (i++; i < infos.size(); i++)
< {
< readers[i].close();
< }
< throw e;
< }
< }
<
< reader = new MultiReader(readers);
< }
< // reader.deletionPolicy = deletionPolicy; It's null anyway
< return reader;
< }
< }.run();
<
< for ( WriterUpdateNotifier notifier : notifiers)
< {
< notifier.onUpdate(this, reader);
< }
< }
---
> void doAfterFlush()
> throws IOException {
> }
2385,2386d2310
< doAfterFlush();
<
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
