Re: Optimize FTS memory footprint

2017-12-12 Thread Michael McCandless
Try upgrading Elasticsearch -- it's up to 6.0 release just a few week ago now -- its (and Lucene's) memory usage has decreased over time. The _uid field in particular will always be costly, unfortunately. Since it's a primary key, every term will be unique, and the term index has to work hard to

Re: Optimize FTS memory footprint

2017-12-12 Thread Michael McCandless
Comments below: On Tue, Nov 28, 2017 at 4:47 PM, elirev wrote: > Thanks Mike . > I did not find any clear way to know it its FST or Norm , or something > else ( unless i miss something ) the fact the FST is an in memory prefix > index lead me to think it using most of the heap . > Our

Re: Optimize FTS memory footprint

2017-12-12 Thread Bingtao Yin
Hi elirev, The field "index" of class "org.apache.lucene.codecs.blocktree.FieldReader" is the fst of each field; its type is FST. I close a index and pick a shard; wirte some code to directly read the shard and then use the reflection to get the actual fst object of _uid field. The ramBytesUsed()