Re: Why does Lucene 7.4.0 commit() Increase Memory Usage x2

2019-04-02 Thread Erick Erickson
Task manager is almost useless for this kind of measurement. You never quite know how much garbage that hasn’t been collected is in that total. You can attach something like jconsole to the running Solr process and hit the “perform full GC” to get a more accurate number. Or you can look at GCVi

Re: Clarification regarding BlockTree implementation of IntersectTermsEnum

2019-04-02 Thread Robert Muir
I think instead of using range queries for custom pagination, it would be best to simply sort by the field normally (with or without searchAfter) On Tue, Apr 2, 2019 at 7:34 AM Stamatis Zampetakis wrote: > > Thanks for the explanation regarding IntersectTermsEnum.next(). I > understand better now

Re: Clarification regarding BlockTree implementation of IntersectTermsEnum

2019-04-02 Thread Stamatis Zampetakis
Thanks for the explanation regarding IntersectTermsEnum.next(). I understand better now. How about indexing a string field containing the firstname and lastname of a person or possibly its address. Values for these fields have variable length and usually they have more than 16 bytes. If my underst

Re: Clarification regarding BlockTree implementation of IntersectTermsEnum

2019-04-02 Thread Robert Muir
Can you explain a little more about your use-case? I think that's the biggest problem here for term range query. Pretty much all range use-cases are converted over to space partitioned data structures (Points) so its unclear why this query would even be used for anything serious. To answer your qu

Re: Lucene migrate to 6.6.5 from 5.5.3

2019-04-02 Thread Adrien Grand
You got it right, you will need to reindex with LongPoint before being about to search with LongPoint#newRangeQuery. It's fine to keep using LegacyNumericRangeQuery until then. On Tue, Apr 2, 2019 at 1:56 PM brahmam wrote: > > Hi, > > Thanks for your response. > > Now that in 6.6.5 it recommends

Re: Clarification regarding BlockTree implementation of IntersectTermsEnum

2019-04-02 Thread Stamatis Zampetakis
Thanks Robert and Uwe for your feedback! I am not sure what you mean that the slowness does not come from the iteration over the term index. I did a small profiling (screenshot attached) of sending repeatedly a TermRangeQuery that matches almost the whole index and I observe that 80% of the time i

Re: Lucene migrate to 6.6.5 from 5.5.3

2019-04-02 Thread brahmam
Hi, Thanks for your response. Now that in 6.6.5 it recommends to use LongPoint.newRangeQuery() instead of deprecated LegacyNumericRangeQuery.newLongRange(), So Does it means, we need to re-index old data using LongPoint so that it is searchable in upgrade scenarios ? In general Lucene supports

Re: Why does Lucene 7.4.0 commit() Increase Memory Usage x2

2019-04-02 Thread thturk
I am watching via task manager. Now i tired to handle this with hard coded way. I create new index and with commit in small index cost low memory. but i dont think that its good way to do this. Its getting harder to manage indexes. -- Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-U

Re: How to control missing value when sorting using *ValuesSource?

2019-04-02 Thread Adrien Grand
+1 On Sun, Mar 31, 2019 at 11:38 PM Michael Sokolov wrote: > > I just want to surface this again in case it got missed; it's a small gap, > but it seems buggy that we can support setMissingValue for SortFields based > on concrete numeric fields, but not for abstract values sources that are > nume

Re: Why does Lucene 7.4.0 commit() Increase Memory Usage x2

2019-04-02 Thread Adrien Grand
How do you measure memory usage? On Mon, Apr 1, 2019 at 8:33 AM thturk wrote: > > Hello, > -For a while i am tring to figure out why ram usage incease x2 than before > after commit one single document. > > -Lucene Version 7.4.0 > -Writer Directory FSDirectory > -Reader Directory MMapDirectory > -

Re: Lucene migrate to 6.6.5 from 5.5.3

2019-04-02 Thread Adrien Grand
Hi Brahmam, This isn't possible: data can only be searched with LongPoint#newRangeQuery if the field has been indexed with LongPoints. On Tue, Apr 2, 2019 at 7:04 AM brahmam wrote: > > Hi, > > In continuation to the previous mail, we see for the second point the newly > added data after upgrade