RE: Program killed

2017-05-18 Thread Uwe Schindler
Hi, The index will stay on old state of the last commit. All data after the last commit is lost, the segments started will be deleted on cleanup when a new merge triggers. This is especially true for sure since Lucene 6 because it uses Java 7's atomic renames to publish commit and also fsyncs t

Re: Program killed

2017-05-18 Thread Kudrettin Güleryüz
Sorry, I was specifically asking for the effect on Lucene index that was being updated when the program is killed during first phase of the two-phase commit model. Would the index be broken? Is there anything that can be done to prevent it, how can it be fixed? Insight on this would be great. Than

Program killed

2017-05-18 Thread Kudrettin Güleryüz
Hi, What happens if a program is killed or if system is rebooted during first phase of the two-phase commit model? Thank you, Kudret

Re: Term Dictionary taking up lots of memory, looking for solutions, lucene 5.3.1

2017-05-18 Thread Uwe Schindler
Hi, Are you sure that the term index is the problem? Even with huge indexes you never need 65 good of heap! That's impossible. Are you sure that your problem is not something else?: - too large heap? Heaps greater than 31 gigs are bad by default. Lucene needs only few heap, although you have lar

Re: Term Dictionary taking up lots of memory, looking for solutions, lucene 5.3.1

2017-05-18 Thread Michael McCandless
That sounds like a fun amount of terms! Note that Lucene does not load all terms into memory; only the "prefix trie", stored as an FST ( http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html), mapping term prefixes to on-disk blocks of terms. FSTs are very compact data str