Hi, > After reading all about the renaming of optimize() and updating my Lucene > libraries to 3.4, I was surprised and confused by what I found. > > I have a 1 segment index (all files are named _1*.*) that had been created > with 3.0.1 code which had been optimized many times (all 3.0.1 code). The > first time my code used the 3.4 libraries with version level set to 3.4 and it tried > to optimize() (still using this now deprecated old call), the new code went wild! > It took up more memory than the heap was limited to, so I believe it is taking > up system resources. We have turned off optimize for now.
Did it throw OutOfMemoryException? I assume not - but I assume, you have seen more virtual memory usage in "top", but that's not really related to optimize/forceMerge. If you are running on a 64 bit platform, Lucene 3.4 will default to MMapDirectory on FSDirectory.open(), which maps the index like a linux/windows swap file into *virtual* memory (not necessary physical memory, but it then directly accesses the file system cache instead of copying to Java heap). This is why "top" mem usage grows, even multiple times of java heap. Nothing to worry about - that's expected and normal (see http://en.wikipedia.org/wiki/Mmap)! > What is it doing? (I believe it is using a TieredMergePolicy, see the IndexWriter > constructor). The merge policy does not have anything to do with that, optimizing or forcemerge to one index segment is the same in all merge policies. > Anyone have any resources to point me to on the subject so I can pick the right > call to forceMerge() and a good MergePolicy before upgrading to Lucene 4.0? > Any ideas? What should be a reasonable 3.4 approach? Are there issues > about the the just complete IndexWriter? Ho about open readers (in the web > app using the index). If you have open readers, many files are open and mapped into virtual memory, so virtual memory usage grows. > -Paul > > p.s. More details available upon request. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org