I opened https://issues.apache.org/jira/browse/LUCENE-5024 to see if we can improve this ...
Mike McCandless http://blog.mikemccandless.com On Thu, May 30, 2013 at 7:31 AM, Michael McCandless <luc...@mikemccandless.com> wrote: > Hi Vitaly, > > This is unfortunately due to a recent change in IndexWriter ... search > for the recent thread "CorruptIndexException when opening Index during > first commit" on this list. > > Normally, if a commit fails for any reason (OS, hardware, JVM crashes) > we simply fall back to the last commit and no intervention is > necessary. > > But if that commit was the very first commit ... we used to try to > detect this and treat it as if there were no index present. This is > the ideal/correct behavior, because "no index present" was in fact the > state of the index before this failed commit. > > However that logic was dangerous and would sometimes result in > IndexWriter thinking no index was present when in fact there was one, > when there were transient errors e.g. due to file descriptor > exhaustion. So to be safe we now throw the CorruptIndexException on > first commit back to the application. > > Your workaround sounds good. Another option instead of the separate > init file, on hitting a CorruptIndexException, might be to list the > directory: if there is only one file, segments_1, then it's a corrupt > first commit and you can recreate the index. > > Mike McCandless > > http://blog.mikemccandless.com > > On Wed, May 29, 2013 at 8:09 PM, Vitaly Funstein <vfunst...@gmail.com> wrote: >> I have encountered a strange issue, that appears to be pretty hard to hit, >> but is still a serious problem when it does occur. It seems that if the JVM >> crashes in a racy fashion with instantiation of IndexWriter, the index may >> be left in an inconsistent state. An attempt to reload such an index on >> restart will result in a CorruptIndexException thrown when calling the >> constructor again. >> >> My idea of a workaround involves the following: >> >> - call commit() immediately after invoking the constructor. >> - create a special init file after the commit to indicate the index was >> created successfully >> - on app startup, check to see if the init file is there; if not, the index >> is corrupt and needs to be recreated. >> >> (The last two already happen in my code) >> >> Is this a valid approach to protect from this sort of race? Is there >> already a built-in mechanism to make constructor fully synchronous wrt >> persisting all the metadata? If not, it seems like there probably should >> be... --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
