Re: Avoid automaton Memory Usage

2013-08-13 Thread Michael McCandless
On Tue, Aug 13, 2013 at 9:44 AM, Anna Björk Nikulásdóttir wrote: > I created these 3 issues for the discussed items: Thanks! If you (or anyone!) want to work up a patch that would be great ... > Thanks a lot for your suggestions (pun intended) ;) ;) Mike McCandless http://blog.mikemccandless

Re: Avoid automaton Memory Usage

2013-08-13 Thread Anna Björk Nikulásdóttir
I created these 3 issues for the discussed items: On disk FST objects: https://issues.apache.org/jira/browse/LUCENE-5174 FuzzySuggester should boost terms with minimal Levenshtein Distance: https://issues.apache.org/jira/browse/LUCENE-5172 AnalyzingSuggester and FuzzySuggester should be able to

Re: Avoid automaton Memory Usage

2013-08-08 Thread Michael McCandless
On Thu, Aug 8, 2013 at 12:54 PM, Anna Björk Nikulásdóttir wrote: > > Am 8.8.2013 um 12:37 schrieb Michael McCandless : > >> >>> What would help in my case as I use the same FST for both analyzers, if the >>> same FST object could be shared among both analyzers. So what I am doing is >>> to use

Re: Avoid automaton Memory Usage

2013-08-08 Thread Anna Björk Nikulásdóttir
Am 8.8.2013 um 12:37 schrieb Michael McCandless : > >> What would help in my case as I use the same FST for both analyzers, if the >> same FST object could be shared among both analyzers. So what I am doing is >> to use AnalyzingSuggester.store() and use the stored file for >> AnalyzingSugges

Re: Avoid automaton Memory Usage

2013-08-08 Thread Michael McCandless
On Wed, Aug 7, 2013 at 1:18 PM, Anna Björk Nikulásdóttir wrote: > Ah I see. I will look into the AnalyzingInfixSuggester. I suppose it could be > useful as an alternative rather to AnalyzingSuggester instead of > FuzzySuggestor ? Yes, but it's very different (it does no fuzzing, and it matches

Re: Avoid automaton Memory Usage

2013-08-07 Thread Anna Björk Nikulásdóttir
Ah I see. I will look into the AnalyzingInfixSuggester. I suppose it could be useful as an alternative rather to AnalyzingSuggester instead of FuzzySuggestor ? What would help in my case as I use the same FST for both analyzers, if the same FST object could be shared among both analyzers. So wh

Re: Avoid automaton Memory Usage

2013-08-07 Thread Michael McCandless
Unfortunately, the FST based suggesters currently must be HEAP resident. In theory this is fixable, e.g. if we could map the FST and then access it via DirectByteBuffer ... maybe open a Jira issue to explore this possibility? You could also try AnalyzingInfixSuggester; it uses a "normal" Lucene i

Avoid automaton Memory Usage

2013-08-07 Thread Anna Björk Nikulásdóttir
Hi, I am using Lucene 4.3 on Android for terms auto suggestions (>500.000). I am using both FuzzySuggester and AnalyzingSuggester, each for their specific strengths. Everything works great but my app consumes 69MB of RAM with most of that dedicated to the suggester classes. This is too much for