Please feel free to provide a pull request that adds the ability to
load the FST off heap to WFSTCompletionLookup. I think it's an
oversight and it'd be a good addition.
Dawid
On Tue, Dec 27, 2022 at 10:35 AM marcos rebelo wrote:
>
> I have the same impression, even if I'm using the MMapDirector
I have the same impression, even if I'm using the MMapDirectory. The data
is on heap.
For my use case, it's a huge waste of memory :( 90% of my data could be
correctly organised and kept in disk.
Thanks for the support
Best regards
Marcos Rebelo
On Tue, 27 Dec 2022, 09:11 Dawid Weiss, wrote:
Looking at the code briefly, I think WFSTCompletionLookup uses on heap
store for the fst. You'd have to load it with off heap fst store instead:
https://github.com/apache/lucene/blob/1b9d98d6ec079e950bdd37137082f81400d3bc2e/lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java
but
Hey all!
I'm loading multiple WFST with ~1.1 Gb and the JVM memory increases
proportionally. Looks like the file is stored in memory, meaning not using
Memory Mapped Files at all.
Example code:
In the following code we setup the Lucene to use /tmp/deleteme2 for the
memory mapped file and we load