Re: Atomicity and AutoCommit

2008-02-27 Thread Michael McCandless
Simon Wistow wrote: On Wed, Feb 27, 2008 at 09:38:55AM -0500, Michael McCandless said: When you previously saw corruption was it due to an OS or machine crash (or power cord got pulled)? If so, you were likely hitting LUCENE-1044, which is fixed on the trunk version of Lucene (to be 2.4 at s

Re: Atomicity and AutoCommit

2008-02-27 Thread Mark Miller
You need to make sure your storage does not lie in response to an fsync command. If it does (most commercial stuff does), you cannot guaranty no corruption. Search google for "your harddrive lies to you" or something. It shouldnt be that hard to take the patch from the issue and apply it to a

Re: Atomicity and AutoCommit

2008-02-27 Thread Simon Wistow
On Wed, Feb 27, 2008 at 09:38:55AM -0500, Michael McCandless said: > > When you previously saw corruption was it due to an OS or machine > crash (or power cord got pulled)? If so, you were likely hitting > LUCENE-1044, which is fixed on the trunk version of Lucene (to be 2.4 > at some point) but

Re: Atomicity and AutoCommit

2008-02-27 Thread Michael McCandless
When you previously saw corruption was it due to an OS or machine crash (or power cord got pulled)? If so, you were likely hitting LUCENE-1044, which is fixed on the trunk version of Lucene (to be 2.4 at some point) but is not fixed in 2.3. If that is what you were hitting, then unfortunately n

Atomicity and AutoCommit

2008-02-27 Thread Simon Wistow
I currently have a set up that indexes into RAM and then periodically merges that into a disk based index. Searches are done from the disk based index and deletes are handled by keeping a list of deleted documents, filtering out search results and applying the deletes to the index at merge tim