Re: deletes when the writer is open and autocommit is set to false

2009-02-11 Thread Vinubalaji Gopal
On Wed, Feb 11, 2009 at 8:50 AM, Michael McCandless wrote: > Hmm -- OK I just fixed that FAQ entry. Thanks for raising this! > Cool. > If you know the doc doesn't exist already, you gain some performance by > using add instead of update. But if performance is already fast enough, it > may be si

Re: deletes when the writer is open and autocommit is set to false

2009-02-11 Thread Michael McCandless
Vinubalaji Gopal wrote: On Wed, Feb 11, 2009 at 2:56 AM, Michael McCandless wrote: IndexWriter can in fact delete documents, by Term or by Query. It also has updateDocument, which under-the-hood simply calls deleteDocuments then addDocument. Awesome that FAQ entry confused me and I did

Re: deletes when the writer is open and autocommit is set to false

2009-02-11 Thread Vinubalaji Gopal
On Wed, Feb 11, 2009 at 2:56 AM, Michael McCandless wrote: > IndexWriter can in fact delete documents, by Term or by Query. It also has > updateDocument, which under-the-hood simply calls deleteDocuments then > addDocument. Awesome that FAQ entry confused me and I didn't look at IndexWriter java

Re: deletes when the writer is open and autocommit is set to false

2009-02-11 Thread Michael McCandless
) and indexing it again. But everytime I have to delete a document - i need to open a new reader since the writer would have already written more documents and I can't reuse the reader for the whole session :( - which makes the operation a bit slow. I thought the better way would be to set autoc

deletes when the writer is open and autocommit is set to false

2009-02-10 Thread Vinubalaji Gopal
whole session :( - which makes the operation a bit slow. I thought the better way would be to set autocommit as false . In which case even if I reopen the reader everytime to delete any document (using IndexReader.deleteDocument(term)) - I wouldn't find it since the commit has not happened. Is

Re: autoCommit

2008-11-10 Thread ChadDavis
That's easy. Thanks. On Mon, Nov 10, 2008 at 1:12 PM, Michael McCandless < [EMAIL PROTECTED]> wrote: > > Actually, all non-deprecated ctors of IndexWriter set autoCommit to false. > Ie, in 3.0 autoCommit false will become the only option. > > Mike > > > C

Re: autoCommit

2008-11-10 Thread Michael McCandless
Actually, all non-deprecated ctors of IndexWriter set autoCommit to false. Ie, in 3.0 autoCommit false will become the only option. Mike ChadDavis wrote: The FAQ's have this index performance tip: Use autoCommit=false when you open your IndexWriter In Lucene 2.3 there are substa

autoCommit

2008-11-10 Thread ChadDavis
The FAQ's have this index performance tip: Use autoCommit=false when you open your IndexWriter > > In Lucene 2.3 there are substantial optimizations for Documents that use > stored fields and term vectors, to save merging of these very large index > files. You should see the be

Re: Atomicity and AutoCommit

2008-02-27 Thread Michael McCandless
Simon Wistow wrote: On Wed, Feb 27, 2008 at 09:38:55AM -0500, Michael McCandless said: When you previously saw corruption was it due to an OS or machine crash (or power cord got pulled)? If so, you were likely hitting LUCENE-1044, which is fixed on the trunk version of Lucene (to be 2.4 at s

Re: Atomicity and AutoCommit

2008-02-27 Thread Mark Miller
You need to make sure your storage does not lie in response to an fsync command. If it does (most commercial stuff does), you cannot guaranty no corruption. Search google for "your harddrive lies to you" or something. It shouldnt be that hard to take the patch from the issue and apply it to a

Re: Atomicity and AutoCommit

2008-02-27 Thread Simon Wistow
On Wed, Feb 27, 2008 at 09:38:55AM -0500, Michael McCandless said: > > When you previously saw corruption was it due to an OS or machine > crash (or power cord got pulled)? If so, you were likely hitting > LUCENE-1044, which is fixed on the trunk version of Lucene (to be 2.4 > at some point) but

Re: Atomicity and AutoCommit

2008-02-27 Thread Michael McCandless
neither buffering updates into RAM nor using autoCommit=false in 2.3 will fully protect you from this issue. Though, both of these approaches should reduce your chance of hitting LUCENE-1044 since they both reduce frequency of commits to the index. Mike Simon Wistow wrote: I currently have a

Atomicity and AutoCommit

2008-02-27 Thread Simon Wistow
play back all actions since the last merge and we're back to where we started. However it occurred to me that this might all be redundant now with Lucene 2.3 (it's possible it might have always been redundant come to think of it) - should I just open a Disk based Index with

corruption issue with 2.3 and autoCommit=false

2008-02-11 Thread Michael McCandless
Heads up! We are working through what looks like an index corruption issue when you use autoCommit=false with IndexWriter, in Lucene 2.3, so please try to avoid doing so if you can... Details are here: https://issues.apache.org/jira/browse/LUCENE-1173 Mike