Re: performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Shai Erera
That's right. In 3x though you have to call addIndexes followed by maybeMerge if you want to achieve the same effect of addindexesNoOptimize. Shai On Friday, November 12, 2010, Marc Sturlese wrote: > > Thanks, so clarifying. As far as I've understood, if I have to end up > optimizing the index j

Re: Deleted File Handles - Index Writer

2010-11-12 Thread Michael McCandless
Are you using compound file format (the default)? If you turn that off (just for this test) do you still see that IndexWriter is holding open the files (35 in your example) after you close all readers? I've found a case, only with compound file, where IndexWriter holds open a SegmentReader on the

Re: performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Marc Sturlese
Thanks, so clarifying. As far as I've understood, if I have to end up optimizing the index just after merging it, no matter if I use the lucene 3.X addIndexes or addIndexesNoOptimize as the sum of time of doing both things will be the same in one case or other. Am I right? -- View this message i

Re: performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Shai Erera
Ok, so a couple of clarifications: addIndexes(Directory...) *does not* trigger any merges. It simply registers the incoming directories in the target index, and returns. You can later call maybeMerge() or optimize() as you see fit. Compound files are irrelevant to addIndexes - it just adds the in

Re: performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Marc Sturlese
Thanks a lot Shai, couple of questions: >> In Lucene 3x there is a new addIndexes which accepts Directory… that >> simply registers the new indexes in the index, without running merges. >> That makes addIndexes very fast. With the lucene 3.X addIndexes which accepts Directory, if after the mer

Re: performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Shai Erera
In Lucene 3x there is a new addIndexes which accepts Directory… that simply registers the new indexes in the index, without running merges. That makes addIndexes very fast. Also, you can consider calling close(false) to not wait for merges. That can speed things up as well. But note that not run

performance merging indexes with addIndexesNoOptimize

2010-11-12 Thread Marc Sturlese
I am doing some test about merge indexing and have a performance doubt I am doing merge in a simple way, something like: FSDirectory indexes[] = new FSDirectory[indexList.size()]; for (int i = 0; i < indexList.size(); i++) { indexes[i] = FSDirectory.open(new File(indexList

Re: IndexTank technology...

2010-11-12 Thread Jorge Handl
> Does anyone know what technology they are using: http://www.indextank.com/ > Is it Lucene under the hood? > Thanks, and apologies for cross-posting. > -Glen I happen to be one of Indextank's developers, so I think I'm qualified to answer your question! ;) We do use a small portion of Lucene, nam

Re: Delete Document from Index. How?

2010-11-12 Thread Pulkit Singhal
Looked at 2.2 api and those methods should be there. So the NoSuchMethodException makes no sense. Are you absolutely sure that your integration between PHP & Java is setup properly and you really are using 2.2? Could there be multiple versions of lucene jars in your classpath? such that older ones

Deleted File Handles - Index Writer

2010-11-12 Thread Thomas Rewig
Hello, I use the searcherManager for LiveIndexing. With watch -n 60 "lsof | grep indexname | grep deleted | wc -l" I see the number of deleted file handles. The number of handles fluctuates during the indexing. 0 -> 35 -> 53 -> 135 -> 40 -> 85 ... Uwe said that this is expected because seg