Re: data corruption in lucene index 2.3.2

2011-10-29 Thread Michael McCandless
Was there any catastrophic event against this index? EG, power loss on the machine, or OS crash, while the app had and IndexWriter open? Mike McCandless http://blog.mikemccandless.com On Fri, Oct 28, 2011 at 4:57 PM, Zhang, Lisheng wrote: > > We are using lucene 2.3.2 (yes we should upgrade) a

Re: IndexWriter loops trying to merge using ConcurrentMergeScheduler

2011-10-29 Thread Michael McCandless
Are you sure SerialMergeScheduler with BalancedMP doesn't loop infinitely (but CMS does)? How many threads are you using for indexing? Mike McCandless http://blog.mikemccandless.com On Thu, Oct 27, 2011 at 8:22 PM, alfredhong wrote: > Hi, Mike, > > Thanks for your analysis.  You are correct in

RE: data corruption in lucene index 2.3.2

2011-10-29 Thread Zhang, Lisheng
Hi Mike, Thanks very much for helps, if indexer was closed but IndexSearcher (therefore IndexReader) was open when there was OS crash, could that cause segment data corruption (I guess it should not)? Best regards, Lisheng -Original Message- From: Michael McCandless [mailto:luc...@mikem

Re: data corruption in lucene index 2.3.2

2011-10-29 Thread Michael McCandless
That's fine -- the reader is read-only and won't corrupt the index if the machine/OS crashes while it's open. Oh, actually: Lucene 2.3.x did not properly fsync files when you closed the IndexWriter (this was fixed in 2.4.0). This means even if you close the writer and a crash occurs the index cou

RE: data corruption in lucene index 2.3.2

2011-10-29 Thread Zhang, Lisheng
Hi Mike, Yes, we had OS crash but it happened minutes after we commited/closed IndexWriter according to our log file. We will consider lucene upgrade. Thanks very much for helps, Lisheng -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Saturday, Octob

Installing on windows.

2011-10-29 Thread patf
Tried a first install on Windows (7 64 bit - but installing as 32 bits) and didn't get very far. Next, at work where I have a linux box, the install was pretty straightforward with the one wrinkle of having to patch setuptools - however it was well explained by messages from the build proce

Installing on windows.

2011-10-29 Thread patf
I added: ./lucene-3.4.0-py2.7-win32.egg to d:\python27\lib\site-packages\easy-install.pth and now lucene imports. Presumably when I try it, it will work. Go figure. Would still though like to know how to build lucene from source using msvc 2008 Pat Tried a first install on Windows (7