Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
I am getting even more confused. I luckily found a copy of one of the corrupted test indices that i had made on 4/28/08...lucky as its the only one I have ever made :) It doesn't have the problem. This is very interesting to me, because the other site that has the problem has been in action for mon

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
Mark Miller wrote: MB: Ah, thanks for clearing the version stuff up...I just assumed that trunk last week was pretty close to 2.3.1. I am def trunk last thurs or fri. Perhaps the problem is after 2.3.1, and perhaps the problem is only with me. OK, thanks for verifying. I'll go ahead and publis

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Sorry, Also, win 2003 java is sun AIX is obviously IBM. On Mon, 2008-05-05 at 18:07 -0400, Michael McCandless wrote: > Mark, > > Which exact version of the JRE are you using? > > Mike > > Mark Miller wrote: > > On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: > >> Actually that sta

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
MB: Ah, thanks for clearing the version stuff up...I just assumed that trunk last week was pretty close to 2.3.1. I am def trunk last thurs or fri. Perhaps the problem is after 2.3.1, and perhaps the problem is only with me. MM: FYI- I upgraded a really old test install (hasnt been touched by a ne

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
Yeah, it's probably confusing, because we currently commit patches to two branches: the trunk (/repos/asf/lucene/java/trunk) and the 2.3 branch (/repos/asf/lucene/java/branches/lucene_2_3). So if you checked out from the trunk, then this is not the 2.3.2 version. The 2.3.2 release candidate is

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Mark, Which exact version of the JRE are you using? Mike Mark Miller wrote: On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: Actually that stack trace looks like it's from trunk, not from 2.3.2 (pre)? OK, I think you said it's from "post 2.3 trunk". Right...the Lucene that sho

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Man, I have even confused myself on these versions at this point. Let me start over. I am having the problem with a version of lucene that was the trunk late last week. Which pretty much means 2.3.2. I'd hate to hold up the release if the problem was only me though. I am trying to work through it

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
On Mon, 2008-05-05 at 17:26 -0400, Michael McCandless wrote: > Actually that stack trace looks like it's from trunk, not from 2.3.2 > (pre)? OK, I think you said it's from "post 2.3 trunk". Right...the Lucene that showed the problem was build from a trunk grab late last week. One of the problem

Re: index corruption with latest lucene

2008-05-05 Thread Michael Busch
If that is the case then I will go ahead and publish the 2.3.2 release? Have you seen this on 2.3.x, Mark? -Michael Michael McCandless wrote: Actually that stack trace looks like it's from trunk, not from 2.3.2(pre)? OK, I think you said it's from "post 2.3 trunk". Another question: is au

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Also, if you can run your tests with assertions enabled, it could catch something... Mike Mark Miller wrote: On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: Hi Mark, Not good! Can you describe how this index was created? Did you use multiple threads on one IndexWriter? Mul

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Actually that stack trace looks like it's from trunk, not from 2.3.2 (pre)? OK, I think you said it's from "post 2.3 trunk". Another question: is autoCommit false or true? More responses below: Mark Miller wrote: On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: Hi Mark, Not g

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
On Mon, 2008-05-05 at 16:32 -0400, Michael McCandless wrote: > Hi Mark, > > Not good! > > Can you describe how this index was created? Did you use multiple > threads on one IndexWriter? Multiple sessions of IndexWriter > appending to the index? addIndexes*? Is the index copied from one

Re: index corruption with latest lucene

2008-05-05 Thread Michael McCandless
Hi Mark, Not good! Can you describe how this index was created? Did you use multiple threads on one IndexWriter? Multiple sessions of IndexWriter appending to the index? addIndexes*? Is the index copied from one place to another after being written and before being searched? If you

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
I thought I remember that thread but couldn't remember the details. I am def getting a similar error, except that I seem to be way more out of bounds than you. I actually get it using an FSDirectory though...and I have gotten it twice already...both upgrades I tried...first on a test machine and

Re: index corruption with latest lucene

2008-05-05 Thread eks dev
hmm, if I am not wrong, it looks awfully similar to the Exception we have seen and concluded it is some black magic with corrupt memory chip or waht-not, but the fact we are not alone makes me wonder now... Subject of this thread was "Strange Exception"... we were able to use this very same inde

Re: index corruption with latest lucene

2008-05-05 Thread Mark Miller
Yeah, its pretty close to 2.3.2, but I think from last week mabye. I finally have one of the stack traces (this comes on the tail complete laptop failure so I am scrambling here) java.lang.IndexOutOfBoundsException: Index: 97, Size: 43 at java.util.ArrayList.RangeCheck(ArrayList.java:572

Re: index corruption with latest lucene

2008-05-05 Thread crspan
coincidence or it is from 2.3.2 ? env: lucene 2.3.2 jdk1.6.0_06 & jdk1.5.0_15 QueryString: illeg^30.820824 technolog^22.290413 transfer^33.307804 Error: java.lang.ArrayIndexOutOfBoundsException: 132704java.lang.ArrayIndexOutOfBoundsException: 132704 at org.apache.lucene.search.BooleanScorer2$

index corruption with latest lucene

2008-05-05 Thread Mark Miller
Last message had a type...I reversed line numbers fields...trying again: Any recent changes that would expose index corruption? I am getting two new errors when trying to search: nullpointer fieldsreaders line 185 indexoutofbounds on fieldinfo line 260 I am kind of screwed, because reindex

index corruption with latest lucene

2008-05-05 Thread Mark Miller
Any recent changes that would expose index corruption? I am getting two new errors when trying to search: nullpointer fieldsreaders line 260 indexoutofbounds on fieldinfo line 185 I am kind of screwed, because reindexing fixes this, but I cant reindex! Any ideas? --

@todo parallelize this one too

2008-05-05 Thread Timo Nentwig
Hi! Unfortunately the search method in ParallelMultiSearcher which is able to take an HitCollector isn't running in parallel and there' even an issue regarding this (LUCENE-990) with zero watchers or votes :-\ So this isn't something that's likely to be done in near future, is it? And question

Re: Search Order

2008-05-05 Thread N. Hira
Please review: http://wiki.apache.org/lucene-java/LuceneFAQ I suspect your question is answered as: How do I make sure that a match in a document title has greater weight than than a match in a document body? -h -- Hira, N.

Re: Search Order

2008-05-05 Thread Erick Erickson
Well, it all depends upon what you mean by problem. Lucene is not returning things in random order, it's returning things in relevance order. There is a lot of discussion about scoring both on the Wiki and in the mail archive if you want to affect the return order. Alternatively, you can sort your

Search Order

2008-05-05 Thread Prashant Saraf
Hi, I am creating a keyword based search engine. Now I am able to search the required data. Now, I notice that the data coming is in random order. Is there any way I can call particular record 1st. I have an example. Suppose I m working on search of Book content, writer, publication. Now if I en