Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
OK I opened: https://issues.apache.org/jira/browse/LUCENE-1331 Mike Paul Taylor wrote: Michael McCandless wrote: Hmmm, you should not close the directory if you are then going to use it to instantiate a searcher. how come it works ? Your code below never closes the searcher? I th

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
It works because Lucene doesn't currently check for it, and, because closing an FSDirectory does not actually make it unusable. In fact it also doesn't catch a double-close call. But it may cause subtle problems, because FSDirectory has this invariant: only a single instance of FSDirecto

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
Michael McCandless wrote: Hmmm, you should not close the directory if you are then going to use it to instantiate a searcher. how come it works ? Your code below never closes the searcher? I think that is most likely the source of your file descriptor leaks. Ok fixed paul --

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
Also, if possible, you should share the IndexSearcher across multiple searches (ie, don't open/close a new one per search). Opening an IndexSearcher can be a resource intensive operation, so you'll see better throughput if you share. (Though in your particular situation it may not matte

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
Hmmm, you should not close the directory if you are then going to use it to instantiate a searcher. Your code below never closes the searcher? I think that is most likely the source of your file descriptor leaks. Mike Paul Taylor wrote: Michael McCandless wrote: Technically you shou

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
Michael McCandless wrote: Technically you should call directory.close() as well, but missing that will not lead to too many open files. How often is that RuntimeException being thrown? EG if a single document is frequently hitting an exception during analysis, your code doesn't close the I

Re: Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Michael McCandless
Technically you should call directory.close() as well, but missing that will not lead to too many open files. How often is that RuntimeException being thrown? EG if a single document is frequently hitting an exception during analysis, your code doesn't close the IndexWriter in that situa

Move from RAMDirectory to FSDirectory causing problem sometimes

2008-07-08 Thread Paul Taylor
Hi, I have been using a RAMDirectory for indexing without any problem, but I then moved to a file based directory to reduce memory usage. this has been working fine on Windows and OSX and my version of linux (redhat) but is failing on a version of linux (archlinux) with 'Too many files opened'