Re: DefaultIndexAccessor

Thanks Mark. I'll wait for your enhancements in IndexAccessor on the new methods. I use mergeFactor = 100. I've read about the merge factor and it's hard to balance both the read/write optimization. What's the number do you use? Thanks again. -vivek On Thu, Feb 28, 2008 at 7:14 PM, Mark Miller <

Re: DefaultIndexAccessor

vivek sar wrote: Mark, Just for my clarification, 1) Would you have indexStop and indexStart methods? If that's the case then I don't have to call close() at all. These new methods would serve as just cleaning up the caches and not closing the thread pool. Yes. This is the approach I agre

Re: DefaultIndexAccessor

Mark, Just for my clarification, 1) Would you have indexStop and indexStart methods? If that's the case then I don't have to call close() at all. These new methods would serve as just cleaning up the caches and not closing the thread pool. I would prefer not to call close() and init() again if

Re: DefaultIndexAccessor

I added the Thread Pool recently, so things did probably work before that. I am certainly willing to put the Thread Pool init in the open call instead of the constructor. As for the best method to use, I was thinking of something along the same lines as what you suggest. One of the decisions

Re: DefaultIndexAccessor

Mark, Yes, I think that's what precisely is happening. I call accessor.close, which shuts down all the ExecutorService. I was assuming the accessor.open would re-open it (I think that's how it worked in older version of your IndexAccessor). Basically, I need a way to stop (or close) all the I

Re: DefaultIndexAccessor

Hey vivek, Sorry you ran into this. I believe the problem is that I had just not foreseen the use case of closing and then reopening the Accessor. The only time I ever close the Accessors is when I am shutting down the JVM. What do you do about all of the IndexAccessor requests while it is in

Re: DefaultIndexAccessor

Mark, Some more information, 1) I run indexwriter every 5 mins 2) After every cycle I check if I need to partition (based on the index size) 3) In the partition interface, a) I first call close on the index accessor (so all the searchers can close before I move tha

Re: DefaultIndexAccessor

Mark, We deployed our indexer (using defaultIndexAccessor) on one of the production site and getting this error, Caused by: java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) at java.util.conc

Re: DefaultIndexAccessor

Mark, Thanks for the quick fix. Actually, it is possible that there might had been simultaneous queries using the MultiSearcher. I assumed it was thread-safe, thus was re-using the same instance. I'll update my application code as well. Thanks, -vivek On Feb 15, 2008 5:56 PM, Mark Miller <[EMAIL

Re: DefaultIndexAccessor

Mark, Here is the scenario when I saw this exception, 1) A search was run which uses MultiSearcher. This search took more than 3 mins to complete (due to index size and multiple indices) 2) Just a minute after the search was started, we started writing (in a separate thread) to one of the index

Re: DefaultIndexAccessor

Here is the fix: https://issues.apache.org/jira/browse/LUCENE-1026 vivek sar wrote: Mark, There seems to be some issue with DefaultMultiIndexAccessor.java. I got following NPE exception, 2008-02-13 07:10:28,021 ERROR [http-7501-Processor6] ReportServiceImpl - java.lang.NullPointerExcep

Re: DefaultIndexAccessor

Okay, sorry about this one vivek. Added to the unit tests to expose this. When I took out the MultiSearcher caching, I kept the concept of sharing a single MultiIndexAccessor. Unfortunately, this meant that multiple threads were sharing the same Searcher to Accessor Map that was used to track w

Re: DefaultIndexAccessor

Hey vivek, sorry to hear you are having problems. I am trying to figure out how you may be seeing this problem. The IndexAccessor cannot return null because you would get an IllegalStateException not a NullPointerException. Also, the released MultiSearcher cannot be null because the Exception

Re: DefaultIndexAccessor

Mark, There seems to be some issue with DefaultMultiIndexAccessor.java. I got following NPE exception, 2008-02-13 07:10:28,021 ERROR [http-7501-Processor6] ReportServiceImpl - java.lang.NullPointerException at org.apache.lucene.indexaccessor.DefaultMultiIndexAccessor.release(Defa

Re: DefaultIndexAccessor

Thanks for your clarifications, Mark! Jay Mark Miller wrote: 5. Although currently IndexSearcher.close() does almost nothing except to close the internal index reader, it might be a safer to close searcher itself as well in closeCachedSearcher(), just in case, the searcher may have other

Re: DefaultIndexAccessor

5. Although currently IndexSearcher.close() does almost nothing except to close the internal index reader, it might be a safer to close searcher itself as well in closeCachedSearcher(), just in case, the searcher may have other resources to release in the future version of Lucene. Didn't c

Re: DefaultIndexAccessor

Thanks for the feedback jay. One at a time: Jay wrote: Great effort for much improved indexaccessor, Mark! A couple questions and observations: 1. In release(Searcher), you removed a check if the given searcher is the cached one from an earlier version. This could potentially cause problems

Re: DefaultIndexAccessor

Great effort for much improved indexaccessor, Mark! A couple questions and observations: 1. In release(Searcher), you removed a check if the given searcher is the cached one from an earlier version. This could potentially cause problems for some people. 2. The createdSearchers variable is not

Re: DefaultIndexAccessor

I replied to the wrong thread -- sorry about that: You still have to be careful if you want to alternate a search and write. If you are loading a lot of docs this way, you would want to hold the Writer to batch the docs, but while you are holding it, you will not have a fresh view of the index

Re: DefaultIndexAccessor

For anyone following this thread who would like to check this out, I put up the new code with the warming capability: https://issues.apache.org/jira/browse/LUCENE-1026 IndexAccessor-02.04.2008.zip

Re: DefaultIndexAccessor

Hello Mark, Thank you for your lengthy and valuable clarification. I have the case - before adding to the index, i must check if a document exist with the same key (actually, double key) - or before deleting a document - I must ensure it exists in the index. Currently I am doing it with my custom

Re: DefaultIndexAccessor

The purpose of IndexAccessor is to coordinate Readers/Writers for a Lucene index. Readers and Writers in Lucene are multi-threaded in that multiple threads may use them at the same time, but they must/should be shared and there are special rules (You cannot delete with a Reader while a Writer i

Re: DefaultIndexAccessor

Hello Mark, I have been reading the code - and honestly I have not understood how it works. I was hoping that this was a solution to the case when you are adding documents - in a multithreaded way, it allows other non-writer threads to be able to see documents added without refreshing the indexsea

Re: DefaultIndexAccessor

IndexAccessor-1.26.2008.zip is the latest one. I will be dating a zip from now on. I hope to post new code with the warming either tonight or tomorrow night. I would be ecstatic to have some help vetting that. Also, I am thinking of making a change so that when you release the Writer the thre