Re: index corruption with latest lucene

2008-05-06 Thread Gopikrishnan Subramani
Thanks Mike. Sorry, I should have mentioned that I'm using 1.6.0_04. I happened to look at the thread a while ago and used -Xbatch but that didn't help which made me think may be it's a different issue. I'll try with -Xint before downgrading to 1.6.0_03 to be doubly sure. -Gopi On 5/6/08, Michae

Re: index corruption with latest lucene

2008-05-06 Thread Michael McCandless
Could you provide more detail on how you hit these two exceptions? Are they reproducible from scratch (creating a new index)? Are you using multiple threads against IndexWriter? Is autoCommit true or false? Any prior exceptions hit? Do your documents have varying number/configuration

Re: index corruption with latest lucene

2008-05-06 Thread Michael McCandless
Are you using JRE 1.6.0_04 or 1.6.0_05? This sounds exactly the same as this: http://www.gossamer-threads.com/lists/lucene/java-user/59650 If it is the same issue, which seems to be a bug in the hotspot compiler, downgrading to JRE 1.6.0_03, or running Java with -Xbatch (forces up-fron

Re: index corruption with latest lucene

2008-05-06 Thread Gopikrishnan Subramani
[ Sorry if I'm hijacking this thread, if you feel this error is unrelated to this thread, I'll move this to a separate thread. ] Even after upgrading to 2.3.1 I'm running into index corruption problems. I'm posting below the exception that is generated while searching. The stack trace looks like,

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
I am getting even more confused. I luckily found a copy of one of the corrupted test indices that i had made on 4/28/08...lucky as its the only one I have ever made :) It doesn't have the problem. This is very interesting to me, because the other site that has the problem has been in action for mon

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
Mark Miller wrote: MB: Ah, thanks for clearing the version stuff up...I just assumed that trunk last week was pretty close to 2.3.1. I am def trunk last thurs or fri. Perhaps the problem is after 2.3.1, and perhaps the problem is only with me. OK, thanks for verifying. I'll go ahead and publis

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Sorry, Also, win 2003 java is sun AIX is obviously IBM. On Mon, 2008-05-05 at 18:07 -0400, Michael McCandless wrote: > Mark, > > Which exact version of the JRE are you using? > > Mike > > Mark Miller wrote: > > On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: > >> Actually that sta

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
MB: Ah, thanks for clearing the version stuff up...I just assumed that trunk last week was pretty close to 2.3.1. I am def trunk last thurs or fri. Perhaps the problem is after 2.3.1, and perhaps the problem is only with me. MM: FYI- I upgraded a really old test install (hasnt been touched by a ne

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
Yeah, it's probably confusing, because we currently commit patches to two branches: the trunk (/repos/asf/lucene/java/trunk) and the 2.3 branch (/repos/asf/lucene/java/branches/lucene_2_3). So if you checked out from the trunk, then this is not the 2.3.2 version. The 2.3.2 release candidate is

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Mark, Which exact version of the JRE are you using? Mike Mark Miller wrote: On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: Actually that stack trace looks like it's from trunk, not from 2.3.2 (pre)? OK, I think you said it's from "post 2.3 trunk". Right...the Lucene that sho

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Man, I have even confused myself on these versions at this point. Let me start over. I am having the problem with a version of lucene that was the trunk late last week. Which pretty much means 2.3.2. I'd hate to hold up the release if the problem was only me though. I am trying to work through it

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: > Actually that stack trace looks like it's from trunk, not from 2.3.2 > (pre)? OK, I think you said it's from "post 2.3 trunk". Right...the Lucene that showed the problem was build from a trunk grab late last week. One of the problem

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
If that is the case then I will go ahead and publish the 2.3.2 release? Have you seen this on 2.3.x, Mark? -Michael Michael McCandless wrote: Actually that stack trace looks like it's from trunk, not from 2.3.2(pre)? OK, I think you said it's from "post 2.3 trunk". Another question: is au

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Also, if you can run your tests with assertions enabled, it could catch something... Mike Mark Miller wrote: On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: Hi Mark, Not good! Can you describe how this index was created? Did you use multiple threads on one IndexWriter? Mul

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Actually that stack trace looks like it's from trunk, not from 2.3.2 (pre)? OK, I think you said it's from "post 2.3 trunk". Another question: is autoCommit false or true? More responses below: Mark Miller wrote: On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: Hi Mark, Not g

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: > Hi Mark, > > Not good! > > Can you describe how this index was created? Did you use multiple > threads on one IndexWriter? Multiple sessions of IndexWriter > appending to the index? addIndexes*? Is the index copied from one

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Hi Mark, Not good! Can you describe how this index was created? Did you use multiple threads on one IndexWriter? Multiple sessions of IndexWriter appending to the index? addIndexes*? Is the index copied from one place to another after being written and before being searched? If you

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
) by just reloading > it from FSDirectory. > > good luck > > > - Original Message > > From: Mark Miller <[EMAIL PROTECTED]> > > To: java-user@lucene.apache.org > > Sent: Monday, 5 May, 2008 9:56:47 PM > > Subject: Re: index corruption with

Re: index corruption with latest lucene

2008-05-05 Thread eks dev
very same index again (it was RAMDirectory loaded from FSDirectory) by just reloading it from FSDirectory. good luck - Original Message > From: Mark Miller <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Monday, 5 May, 2008 9:56:47 PM > Subject: Re: index c

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Yeah, its pretty close to 2.3.2, but I think from last week mabye. I finally have one of the stack traces (this comes on the tail complete laptop failure so I am scrambling here) java.lang.IndexOutOfBoundsException: Index: 97, Size: 43 at java.util.ArrayList.RangeCheck(ArrayList.java:572

Re: index corruption with latest lucene

2008-05-05 Thread crspan
coincidence or it is from 2.3.2 ? env: lucene 2.3.2 jdk1.6.0_06 & jdk1.5.0_15 QueryString: illeg^30.820824 technolog^22.290413 transfer^33.307804 Error: java.lang.ArrayIndexOutOfBoundsException: 132704java.lang.ArrayIndexOutOfBoundsException: 132704 at org.apache.lucene.search.BooleanScorer2$

index corruption with latest lucene

2008-05-05 Thread Mark Miller
Last message had a type...I reversed line numbers fields...trying again: Any recent changes that would expose index corruption? I am getting two new errors when trying to search: nullpointer fieldsreaders line 185 indexoutofbounds on fieldinfo line 260 I am kind of screwed, because reindex

index corruption with latest lucene

2008-05-05 Thread Mark Miller
Any recent changes that would expose index corruption? I am getting two new errors when trying to search: nullpointer fieldsreaders line 260 indexoutofbounds on fieldinfo line 185 I am kind of screwed, because reindexing fixes this, but I cant reindex! Any ideas? --