回复: 回复:RE: why lucene not use DirectByteBuffer in NIOFSDirectory

2013-08-07 Thread wangzhijiang999
Hi Uwe,    Thank you for your quick reply. I pasted all the snapshot pictures of "top","jstat","jmap","free" commands results in the tests of last email,  but these pictures seemed not to be shown.     Your explainations already solved all my confusings.   - 原邮件 - 发件人:

Re: WeakIdentityMap high memory usage

2013-08-07 Thread Denis Bazhenov
Uwe, Michael, thank you very much for your help. We have deployed one of the nodes in our system and tomorrow I'll have more information on that, but it seems that setUseUnmap(false) trick did the job. RT drops significantly comparing to 3.6.0 version. We have about 100 rps per search-node and c

Re-load Suggester question

2013-08-07 Thread Brent Montrose
Hello I've recently implemented a FuzzySuggester and an AnalyingInFixSuggester with an web app. It works great, and performs well. I'd like to be able to reload or re-build the underlying indexes with an updated dictionary periodically throughout the day without having to restart the app. I've

RE: WeakIdentityMap high memory usage

2013-08-07 Thread Uwe Schindler
Hi Denis, I assume you are using Lucene 3.6.0, because in Lucene 3.6.1 the tracking of buffers using weak references is also done (although you cannot switch it off, unfortunately). I can confirm what Mike says: Its all weak references and the overhead is maybe large, but it gets freed when me

Re: Avoid automaton Memory Usage

2013-08-07 Thread Anna Björk Nikulásdóttir
Ah I see. I will look into the AnalyzingInfixSuggester. I suppose it could be useful as an alternative rather to AnalyzingSuggester instead of FuzzySuggestor ? What would help in my case as I use the same FST for both analyzers, if the same FST object could be shared among both analyzers. So wh

RE: 回复:RE: why lucene not use DirectByteBuffer in NIOFSDirectory

2013-08-07 Thread Uwe Schindler
Hi, 1) ByteBuffer bb=ByteBuffer.wrap(new byte[len]); //bb is HeapByteBuffer channel.read(bb); this takes indeed 2 Gig of heap because you allocate 2 Gigabytes on java heap. The additional direct memory comes from JVM internals. This depends on the JVM you are using: Some use mmapin

回复:RE: why lucene not use DirectByteBuffer in NIOFSDirectory

2013-08-07 Thread wangzhijiang999
Hi Uwe:   Thank you for your detail explaination and I learnt a lot from your message.   First, the direct buffer and the FS cache do not share the same memory areas.   Second, accessing direct memory is slower than accessing heap memory from java code.   In addition, I tested t

Re: Avoid automaton Memory Usage

2013-08-07 Thread Michael McCandless
Unfortunately, the FST based suggesters currently must be HEAP resident. In theory this is fixable, e.g. if we could map the FST and then access it via DirectByteBuffer ... maybe open a Jira issue to explore this possibility? You could also try AnalyzingInfixSuggester; it uses a "normal" Lucene i

Avoid automaton Memory Usage

2013-08-07 Thread Anna Björk Nikulásdóttir
Hi, I am using Lucene 4.3 on Android for terms auto suggestions (>500.000). I am using both FuzzySuggester and AnalyzingSuggester, each for their specific strengths. Everything works great but my app consumes 69MB of RAM with most of that dedicated to the suggester classes. This is too much for

How to Phrase Search Query with wild card

2013-08-07 Thread TechRay
(Please note that I am very new to lucene/hibernate and any sort of database usage that isn't straightforward SQL queries) I am using hibernate with Lucene and I am trying to do a search query that when "Foo B" is searched it would display records with say a 'title' of "Foo Bar", a 'description' o

Re: WeakIdentityMap high memory usage

2013-08-07 Thread Michael McCandless
This map is used to track all cloned open files, which can be a very large number over time (each search will create maybe 3 of them). This is done as a "best effort" to prevent SEGV (JVM dies) if you accidentally try to use an IndexReader after it was closed, while using MMapDirectory. However,

Re: SortField is not serializable

2013-08-07 Thread Robert Muir
Lucene 4 does not support serialization. On Wed, Aug 7, 2013 at 4:52 AM, Yonghui Zhao wrote: > In lucene 4.3, SortField is not serializable now. > > When I try to serialize a request which has SortField, > java.io.NotSerializableException: org.apache.lucene.search.SortField > exception is thrown

Re: Searching within a Search Result

2013-08-07 Thread Ian Lea
Your BooleanQuery code looks fine. Do the label and abstract searches work independently? Are there docs that match both searches? There are tips on non-working searches in the FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_no_hits_.2F_incorrect_hits.3F If that doesn't help

SortField is not serializable

2013-08-07 Thread Yonghui Zhao
In lucene 4.3, SortField is not serializable now. When I try to serialize a request which has SortField, java.io.NotSerializableException: org.apache.lucene.search.SortField exception is thrown out. Any work around?

WeakIdentityMap high memory usage

2013-08-07 Thread Denis Bazhenov
We have upgraded from Lucene 3.6 to 4.4.On the production we faced high minor GC time. Heap dump showed that one of the biggest objects by size is org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference. About 11 million instances with about 377 megabytes of memory in total (this is not eve