Re: Autocompletion on large index

2011-07-07 Thread Dawid Weiss
You can actually make an (relatively easy) change to FSTLookup to allow infix matches (or word-boundary matches). This should have little impact on memory and nearly zero on performance. This issue is tracking this: https://issues.apache.org/jira/browse/SOLR-2479 I should have implemented it a wh

Re: Autocompletion on large index

2011-07-07 Thread Michael McCandless
On Thu, Jul 7, 2011 at 7:00 AM, Dawid Weiss wrote: > Another option to tradeoff dize and mem is to do a lru like cache of suffix > nodes/ registry. Im still working on that api replacement patch so any > changes to fst right now scare me... That sounds cool too! I opened LUCENE-3289 to allow con

Re: Autocompletion on large index

2011-07-07 Thread Elmer
> Elmer > > > > > > -Oorspronkelijk bericht----- > > From: Michael McCandless > > Sent: Wednesday, July 06, 2011 8:39 PM > > To: java-user@lucene.apache.org > > Subject: Re: Autocompletion on large index > > > > Hmm... so I suspect the f

Re: Autocompletion on large index

2011-07-07 Thread Elmer
esulted in heap > space problems. That's why I was interested in using the new suggest classes > :) > > BR, > Elmer > > -Oorspronkelijk bericht- > From: Michael McCandless > Sent: Wednesday, July 06, 2011 6:23 PM > To: java-user@lucene.apache.org > S

Re: Autocompletion on large index

2011-07-07 Thread Elmer
: Michael McCandless > Sent: Wednesday, July 06, 2011 8:39 PM > To: java-user@lucene.apache.org > Subject: Re: Autocompletion on large index > > Hmm... so I suspect the fst suggest module must first gather up all > titles, then sort them, in RAM, and then build the actual F

Re: Autocompletion on large index

2011-07-07 Thread Dawid Weiss
gt; I'll test further tomorrow and report on mem usage for runnable smaller >>> indexes. >>> I will email you privately for sharing the index to work with. >>> >>> BR, >>> Elmer >>> >>> >>> -Oorspronkelijk bericht- >>

Re: Autocompletion on large index

2011-07-07 Thread Michael McCandless
>> BR, >> Elmer >> >> >> -Oorspronkelijk bericht----- >> From: Michael McCandless >> Sent: Wednesday, July 06, 2011 8:39 PM >> To: java-user@lucene.apache.org >> Subject: Re: Autocompletion on large index >> >> Hmm... so I sus

Re: Autocompletion on large index

2011-07-07 Thread Dawid Weiss
-- > From: Michael McCandless > Sent: Wednesday, July 06, 2011 8:39 PM > To: java-user@lucene.apache.org > Subject: Re: Autocompletion on large index > > Hmm... so I suspect the fst suggest module must first gather up all > titles, then sort them, in RAM, and then build the act

Re: Autocompletion on large index

2011-07-06 Thread Elmer
chael McCandless Sent: Wednesday, July 06, 2011 8:39 PM To: java-user@lucene.apache.org Subject: Re: Autocompletion on large index Hmm... so I suspect the fst suggest module must first gather up all titles, then sort them, in RAM, and then build the actual FST. Maybe it's this gather + sort tha

Re: Autocompletion on large index

2011-07-06 Thread Michael McCandless
arger set of publication titles is > currently no-go with 2.5GB of heapspace, only having a main class that > builds the LookUp data. > > BR, > Elmer > > > -Oorspronkelijk bericht- From: Michael McCandless > Sent: Wednesday, July 06, 2011 6:23 PM > To: java-use

Re: Autocompletion on large index

2011-07-06 Thread Elmer
dless Sent: Wednesday, July 06, 2011 6:23 PM To: java-user@lucene.apache.org Subject: Re: Autocompletion on large index You could try storing your autocomplete index in a RAMDirectory? But: I'm surprised you see the FST suggest impl using up so much RAM; very low memory usage is one of the

Re: Autocompletion on large index

2011-07-06 Thread Elmer
Up data. BR, Elmer -Oorspronkelijk bericht- From: Michael McCandless Sent: Wednesday, July 06, 2011 6:23 PM To: java-user@lucene.apache.org Subject: Re: Autocompletion on large index You could try storing your autocomplete index in a RAMDirectory? But: I'm surprised you see the FST suggest

Re: Autocompletion on large index

2011-07-06 Thread Michael McCandless
You could try storing your autocomplete index in a RAMDirectory? But: I'm surprised you see the FST suggest impl using up so much RAM; very low memory usage is one of the strengths of the FST approach. Can you share the text (titles) you are feeding to the suggest module? Mike McCandless http://

Autocompletion on large index

2011-07-06 Thread Elmer
Hi again. I have created my own autocompleter based on the spellchecker. This works well in a sense that it is able to create an auto completion index from my 'publication' index. However, integrated in my web application, each keypress asks autocompleter to search the index, which is stored on di