Re: CorruptIndexException when opening Index during first commit

2013-05-30 Thread Michael McCandless
OK I opened https://issues.apache.org/jira/browse/LUCENE-5024 ... Geoff can you describe your idea there? Thanks. Mike McCandless http://blog.mikemccandless.com On Thu, May 30, 2013 at 7:35 AM, Michael McCandless wrote: > On Mon, May 20, 2013 at 9:22 AM, Geoff Cooney wrote: >>> The problem

Re: CorruptIndexException when opening Index during first commit

2013-05-30 Thread Michael McCandless
On Mon, May 20, 2013 at 9:22 AM, Geoff Cooney wrote: >> The problem is we can't reliably differentiate commit-in-progress from >> a corrupt first commit... > > I think you can tell them apart with high probability because the checksum > is off by exactly one(at least in lucene 3.5 where I'm lookin

Re: CorruptIndexException when opening Index during first commit

2013-05-20 Thread Geoff Cooney
> The problem is we can't reliably differentiate commit-in-progress from > a corrupt first commit... I think you can tell them apart with high probability because the checksum is off by exactly one(at least in lucene 3.5 where I'm looking). It does seem dangerous to rely on an implementation det

Re: CorruptIndexException when opening Index during first commit

2013-05-17 Thread Michael McCandless
On Thu, May 16, 2013 at 2:59 PM, Geoff Cooney wrote: > Thanks for the response, Mike. > > If I understand correctly, the problem was incorrectly identifying a large > corrupted index as a non-existant index? Actually, a large healthy index as non-existent (because of file descriptor exhaustion).

Re: CorruptIndexException when opening Index during first commit

2013-05-16 Thread Geoff Cooney
Thanks for the response, Mike. If I understand correctly, the problem was incorrectly identifying a large corrupted index as a non-existant index? It seems like you'd really want an index with first-commit in progress to behave like an index with zero documents, as opposed to a non-existant inde

Re: CorruptIndexException when opening Index during first commit

2013-05-16 Thread Michael McCandless
Unfortunately this is expected behavior. We tried to fix it in LUCENE-2812, but this fix was too dangerous and could sometimes erase a good index (if transient IOExcs are happening, e.g. due to file descriptor exhaustion) so we reverted back in LUCENE-4738, so that indexExists will return true, an