Lucene segment selection strategy

2015-07-17 Thread Geoff Cooney
Hi, We recently had an issue with an index where two sequential aborted but unsuccessfully rolled back commits resulted in empty segments_n files, segments_i13p and segments_i13q in this case. This resulted in an exception whenever we tried to open the index until we manually removed the bad segm

Custom Segment Element

2013-10-28 Thread Geoff Cooney
Hi, We build a custom parallel index for geo fields containing a z-order based tree structure. This was originally developed against 3.x but we are looking at upgrading to lucene 4.x now. In the current implementation, we inject ourselves into the IndexingChain on indexing. The challenge is on me

Re: CorruptIndexException when opening Index during first commit

2013-05-20 Thread Geoff Cooney
> The problem is we can't reliably differentiate commit-in-progress from > a corrupt first commit... I think you can tell them apart with high probability because the checksum is off by exactly one(at least in lucene 3.5 where I'm looking). It does seem dangerous to rely on an implementation det

Re: CorruptIndexException when opening Index during first commit

2013-05-16 Thread Geoff Cooney
n > LUCENE-4738, so that indexExists will return true, and trying to open > an IndexReader will throw CorruptIndexException, when the first-commit > has started but not yet finished. > > Mike McCandless > > http://blog.mikemccandless.com > > > On Thu, May 16, 2013

CorruptIndexException when opening Index during first commit

2013-05-16 Thread Geoff Cooney
Hi, We're occasionally seeing a CorruptIndexException when a searcher is opened on a new index. When we see the exception, it looks like what is happening is that the searcher is opening the index after prepareCommit for segments_1 but before the commit is completed. Because there is no prior co

Re: content disappears in the index

2012-11-13 Thread Geoff Cooney
Hi, I've been following this thread and happen to have a simple TruncatingFilter class I wrote for the same purpose. I think this should do what you want: import java.io.IOException; import org.apache.lucene.analysis.TokenFilter; import org.apache.lucene.analysis.TokenStream; import org.apach

Re: forcing an IndexWriter to close

2012-06-04 Thread Geoff Cooney
as much as possible. It's a > dangerous method. > > Shai > > On Mon, Jun 4, 2012 at 4:59 PM, Geoff Cooney > wrote: > > > Hi, > > > > Is there a safe way to forcefully close an IndexWriter that is unable to > > flush to disk? We're see

forcing an IndexWriter to close

2012-06-04 Thread Geoff Cooney
Hi, Is there a safe way to forcefully close an IndexWriter that is unable to flush to disk? We're seeing occasional issues where an IndexWriter encounters an IOException on close and does not release the write lock. The IndexWriter documentation lists this as desired behavior so that clients can