Re: CorruptIndexException when opening Index during first commit

2013-05-30 Thread Michael McCandless
OK I opened https://issues.apache.org/jira/browse/LUCENE-5024 ... Geoff can you describe your idea there? Thanks. Mike McCandless http://blog.mikemccandless.com On Thu, May 30, 2013 at 7:35 AM, Michael McCandless wrote: > On Mon, May 20, 2013 at 9:22 AM, Geoff Cooney wrote: >>> The problem

Re: CorruptIndexException when opening Index during first commit

2013-05-30 Thread Michael McCandless
On Mon, May 20, 2013 at 9:22 AM, Geoff Cooney wrote: >> The problem is we can't reliably differentiate commit-in-progress from >> a corrupt first commit... > > I think you can tell them apart with high probability because the checksum > is off by exactly one(at least in lucene 3.5 where I'm lookin

Re: CorruptIndexException when opening Index during first commit

2013-05-20 Thread Geoff Cooney
> The problem is we can't reliably differentiate commit-in-progress from > a corrupt first commit... I think you can tell them apart with high probability because the checksum is off by exactly one(at least in lucene 3.5 where I'm looking). It does seem dangerous to rely on an implementation det

Re: CorruptIndexException when opening Index during first commit

2013-05-17 Thread Michael McCandless
On Thu, May 16, 2013 at 2:59 PM, Geoff Cooney wrote: > Thanks for the response, Mike. > > If I understand correctly, the problem was incorrectly identifying a large > corrupted index as a non-existant index? Actually, a large healthy index as non-existent (because of file descriptor exhaustion).

Re: CorruptIndexException when opening Index during first commit

2013-05-16 Thread Geoff Cooney
Thanks for the response, Mike. If I understand correctly, the problem was incorrectly identifying a large corrupted index as a non-existant index? It seems like you'd really want an index with first-commit in progress to behave like an index with zero documents, as opposed to a non-existant inde

Re: CorruptIndexException when opening Index during first commit

2013-05-16 Thread Michael McCandless
Unfortunately this is expected behavior. We tried to fix it in LUCENE-2812, but this fix was too dangerous and could sometimes erase a good index (if transient IOExcs are happening, e.g. due to file descriptor exhaustion) so we reverted back in LUCENE-4738, so that indexExists will return true, an

CorruptIndexException when opening Index during first commit

2013-05-16 Thread Geoff Cooney
Hi, We're occasionally seeing a CorruptIndexException when a searcher is opened on a new index. When we see the exception, it looks like what is happening is that the searcher is opening the index after prepareCommit for segments_1 but before the commit is completed. Because there is no prior co