Re: TermScorer default buffer size

On Friday 09 January 2009 05:29:15 John Wang wrote: > Makes sense. > I didn't think 32 was the empirically determined magic number ;) That number does have a history, but I don't know the details. > Are you planning to do a patch for this? No, but could you open an issue and mention the perform

Re: Writer memory released after flush

I am sharding my index database and as i move on to different database, the memory acquired by the writer is not released and shortly the app is crashed due to out of memory. During indexing a particular folder, i want the RAM buffer set to 50MB, thereafter i should do only deletions and the R

Re: TermScorer default buffer size

Makes sense. I didn't think 32 was the empirically determined magic number ;) Are you planning to do a patch for this? -John On Thu, Jan 8, 2009 at 1:27 AM, Paul Elschot wrote: > John, > > Continuing, see below. > > On Wednesday 07 January 2009 14:24:15 Paul Elschot wrote: > > On Wednesday 07

Re: IndexSearcher - architecture - shortest possible latency between update of index (via IndexWriter/IndexReader) and querying the same using IndexSearcher

Thanks Erick for the clarifications regarding the same. Assuming we have a RAMDirectory based inverted index (along with a FSDirectory for a secondary storage index) - what would be the limitation on the RAMDirectory capacity in terms of the size of the index. (other than the main memory i.e.

Re: IndexSearcher - architecture - shortest possible latency between update of index (via IndexWriter/IndexReader) and querying the same using IndexSearcher

This topic has been discussed *very* extensively, so I'd recommend you search the mail archive (see http://wiki.apache.org/lucene-java/MailingListArchives ) since there are more good ideas there than I can remember. But the short answer is that you must open a new searcher for modifications to be s

IndexSearcher - architecture - shortest possible latency between update of index (via IndexWriter/IndexReader) and querying the same using IndexSearcher

Hi- For one of our apps - we are doing a lot of additions and deletions (high frequency) at any given time. Assuming the same index directory under discussion between the writers ( IndexWriter and IndexReader, the latter for deletions) and the readers (IndexSearcher to begin with) - we want

Re: Help with installing Lucene

Please start a new topic when changing subjects. See: http://people.apache.org/~hossman/#threadhijack Thread Hijacking on Mailing Lists When starting a new discussion on a mailing list, please do not reply to an existing message, instead start a

Re: Help with installing Lucene

Hello, I came across some new information regarding the original architecture. We have a file on a website that basically contains all the links of all the articles that are searchable. This file is meant to be a crawler starting point. The articles already have metadata that can be used for inde

Re: Help with installing Lucene

On Jan 7, 2009, at 12:42 PM, ahammad wrote: Greg Shackles wrote: Depending on what you need, there might be something already built that can do what you want. I can't look up links right now but you might want to look into Solr and see if that works for what you want. Otherwise, I

Re: Writer memory released after flush

Opening a writer should be a very fast operation, but closing may be costly since that does a commit() to make the changes permanent in the IO system. Still... if you don't need to refresh your searchers you may want to just leave the writers open, but perhaps lower the RAM buffer if you

Re: Writer memory released after flush

Whether it will be a good idea to open writer only when required. Whether opening writer will take some time as like searcher? Whether to initialize writer will take some time? Regards Ganesh - Original Message - From: "Michael McCandless" To: Sent: Thursday, January 08, 2009 3:5

Re: Writer memory released after flush

IndexWriter holds onto the memory & recycles it. It's best to close the writer if you need to release the memory. Mike Ganesh wrote: I am maintaing multiple indexes and all writers will be opened. Based on some criteria, the document is added to a particular index. I am using 50 MB of

Re: TermScorer default buffer size

John, Continuing, see below. On Wednesday 07 January 2009 14:24:15 Paul Elschot wrote: > On Wednesday 07 January 2009 07:25:17 John Wang wrote: > > Hi: > > > >The default buffer size (for docid,score etc) is 32 in TermScorer. > > > > We have a large index with some terms to have very d

Writer memory released after flush

I am maintaing multiple indexes and all writers will be opened. Based on some criteria, the document is added to a particular index. I am using 50 MB of RAM buffer size. Whether the buffer will be released once flush or optinization done or the writer will maintain the memory and recyle it. Re