Cache full text into memory

2010-07-13 Thread Li Li
I want to cache full text into memory to improve performance. Full text is only used to highlight in my application(But it's very time consuming, My avg query time is about 250ms, I guess it will cost about 50ms if I just get top 10 full text. Things get worse when get more full text because i

Re: Continuously iterate over documents in index

2010-07-13 Thread Shashi Kant
On Tue, Jul 13, 2010 at 5:17 PM, Max Lynch wrote: > Hi, > I would like to continuously iterate over the documents in my lucene index > as the index is updated.  Kind of like a "stream" of documents.  Is there a > way I can achieve this? > > Would something like this be sufficient (untested): > >  

How to create a fuzzy suggest

2010-07-13 Thread Kai Weingärtner
Hello, I am trying to create a suggest search (search results are displayed while the user is entering the query) for names, but the search should also give results if the given name just sounds like an indexed name. However a perfect match should be ranked higher than a similar sounding matc

Re: Best practices for searcher memory usage?

2010-07-13 Thread Paul Libbrecht
Le 13-juil.-10 à 23:49, Christopher Condit a écrit : * are there performance optimizations that I haven't thought of? The first and most important one I'd think of is get rid of NFS. You can happily do a local copy which might, even for 10 Gb take less than 30 seconds at server start. pa

Best practices for searcher memory usage?

2010-07-13 Thread Christopher Condit
We're getting up there in terms of corpus size for our Lucene indexing application: * 20 million documents * all fields need to be stored * 10 short fields / document * 1 long free text field / document (analyzed with a custom shingle-based analyzer) * 140GB total index size * Optimized into a s

Continuously iterate over documents in index

2010-07-13 Thread Max Lynch
Hi, I would like to continuously iterate over the documents in my lucene index as the index is updated. Kind of like a "stream" of documents. Is there a way I can achieve this? Would something like this be sufficient (untested): int currentDocId = 0; while(true) { for(; currentDocId < r

Re: MaxFieldLength

2010-07-13 Thread Erick Erickson
Yep, that's pretty much it.. Erick On Tue, Jul 13, 2010 at 12:47 AM, manjula wijewickrema wrote: > Ok Erick, answer is there. If there is no any document exceeds the default > maxfieldlength, then no any document will be truncated although we increase > the no. of documents in the index. A'm I c

RE: ShingleFilter failing with more terms than index phrase

2010-07-13 Thread Steven A Rowe
Hi Ethan, You'll probably get better answers about Solr specific stuff on the solr-u...@a.l.o list. Check out PositionFilterFactory - it may address your issue: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.PositionFilterFactory Steve > -Original Message- > From: Et

ShingleFilter failing with more terms than index phrase

2010-07-13 Thread Ethan Collins
I am using lucene 2.9.3 (via Solr 1.4.1) on windows and am trying to understand ShingleFilter. I wrote the following code and find that if I provide more words than the actual phrase indexed in the field, then the search on that field fails (no score found with debugQuery=true). Here is an example