RE: Applying LUCENE-1606 -- which version

2010-01-26 Thread Sriram Muthuswamy Chittathoor
I tried on the flex brach and it seems to be working well. This is the general pattern I am following (both for prefix queries and queries of the sort ".{0,3}poker.*". I am assuming I could just use this as part of a larger BooleanQuery. AutomatonQuery aq = new RegexpQuery(newTerm("123.*pok

Re: file open handles?

2010-01-26 Thread Jake Mannix
On Tue, Jan 26, 2010 at 11:13 PM, Jamie wrote: > > Hi Jake > > Thanks for the info. Are you specifically referring to > http://issues.apache.org/jira/browse/LUCENE-2120? > Yep, that's the issue I'm referring to. > Our app indexes about 170 50k documents per second in heavy load. In any > case,

Re: file open handles?

2010-01-26 Thread Jamie
Hi Jake Thanks for the info. Are you specifically referring to http://issues.apache.org/jira/browse/LUCENE-2120? Our app indexes about 170 50k documents per second in heavy load. In any case, I doubt it is the realtime IndexReader issue since when we ran the test, no searching was taking plac

Re: file open handles?

2010-01-26 Thread Jake Mannix
Hi Jamie, How fast are you indexing (number of documents per second)? We also ran into this when trying to perf test heavy query throughput while doing rapid indexing under exactly these conditions: call getReader() every time a search is executed (so that it's "really real time"). The answe

Re: file open handles?

2010-01-26 Thread Jamie
Hi Jason I am calling it each time the search takes place. It is no only these files, there are more. In fact, the number of files increases quite frequently. I am seriously worried that we will run out of file handles after a period of time. I am calling getReader every time a search takes p

Re: file open handles?

2010-01-26 Thread Jason Rutherglen
Jamie, How often are you calling getReader? Is it only these files? Jason On Tue, Jan 26, 2010 at 12:58 PM, Jamie wrote: > Ok. I spoke too soon. The problem is not solved. I am still seeing these > file handles lying around. Is this something I should be worried about? > We are now closing the

Re: file open handles?

2010-01-26 Thread Jamie
Ok. I spoke too soon. The problem is not solved. I am still seeing these file handles lying around. Is this something I should be worried about? We are now closing the IndexReader but the IndexWriter remains open through out the running of the program. problem is not solved s# lsof | grep index

Re: file open handles?

2010-01-26 Thread Jamie
HI Jason Thanks a ton. Problem solved. No more stray file handles! Jamie On 2010/01/26 10:03 PM, Jason Rutherglen wrote: You can call close on the reader obtained via writer.getReader. Well, actually, you'll need to. :) The underlying writer will not be affected though. On Tue, Jan 26, 201

Re: file open handles?

2010-01-26 Thread Jason Rutherglen
You can call close on the reader obtained via writer.getReader. Well, actually, you'll need to. :) The underlying writer will not be affected though. On Tue, Jan 26, 2010 at 11:45 AM, Jamie wrote: > Hi Jason > > No .I wasn't sure whether I needed to or not. We have just switched over to > usin

Re: file open handles?

2010-01-26 Thread Jamie
Hi Jason No .I wasn't sure whether I needed to or not. We have just switched over to using the the writer.getReader() method and was worried if I closed the Reader that the Writer would be closed too. Is this misguided? Jamie On 2010/01/26 09:40 PM, Jason Rutherglen wrote: Jamie, Are you

Re: file open handles?

2010-01-26 Thread Jason Rutherglen
Jamie, Are you calling close on the reader? Jason On Tue, Jan 26, 2010 at 11:23 AM, Jamie wrote: > Hi Erick > > Our app is a long running server. Is it a problem if indexes are never > closed? Our searchers > do see the latest snapshot as we use writer.getReader() method for fast > searches. >

Re: file open handles?

2010-01-26 Thread Jamie
Hi Erick Our app is a long running server. Is it a problem if indexes are never closed? Our searchers do see the latest snapshot as we use writer.getReader() method for fast searches. In our case, the index is written to continously, and there are many clients reading from the index simultaneo

Re: file open handles?

2010-01-26 Thread Erick Erickson
They should disappear when the last program holding them open closes them or terminates. So I'm guessing that you have a searcher or some such still accessing the files. This is why your searcher doesn't see new additions until you reopen it, the "snapshot" the literature talks about is just the ol

file open handles?

2010-01-26 Thread Jamie
Hi There I am sure if this is anything to worry about, but I thought I'd ask just in case. We're trying to get a handle on our file handles, so to speak. When I type the following: r...@ubuntu:~# lsof | grep java I get a whole bunch of Index files created by Lucene. Are these file handles

Average Precision - TREC-3

2010-01-26 Thread Ivan Provalov
We are looking into making some improvements to relevance ranking of our search platform based on Lucene. We started by running the Ad Hoc TREC task on the TREC-3 data using "out-of-the-box" Lucene. The reason to run this old TREC-3 (TIPSTER Disk 1 and Disk 2; topics 151-200) data was that the