I committed another change to 3.x's CheckIndex (to also invoke
.nextPosition); can you run that and see if it can detect this?

Is there any way I can get this index?

One question: how large are your payloads, typically?

Does the exception still occur on the index fully rebuilt with 3.1?

Mike

http://blog.mikemccandless.com

2011/4/6 袁武 [GMail] <yuanwu.m...@gmail.com>:
> Dear Mike:
>
> I have run the CheckIndex of branch_3x, and the result report is listed
> below:
>
> [oracle@server bin]$ java -classpath ./ org.apache.lucene.index.CheckIndex 
> /data/Index/URL/Generic/  -fix
> NOTE: testing will be more thorough if you run java with 
> '-ea:org.apache.lucene...', so assertions are enabled
> Opening index @ /data/Index/URL/Generic/
> Segments file=segments_ayup numSegments=1 version=FORMAT_DIAGNOSTICS [Lucene 
> 2.9]
>   1 of 1: name=_bym0 docCount=1344278
>     compound=false
>     hasProx=true
>     numFiles=11
>     size (MB)=15,979.593
>     diagnostics = {optimize=true, mergeFactor=3, 
> os.version=2.6.27.41-170.2.117.                                               
>               fc10.x86_64, os=Linux, mergeDocStores=true, 
> lucene.version=3.0-dev, source=merge                                          
>                    , os.arch=amd64, java.version=1.6.0_21, java.vendor=Sun 
> Microsystems Inc.}
>     no deletions
>     test: open reader.........OK
>     test: fields..............OK [11 fields]
>     test: field norms.........OK [2 fields]
>     test: terms, freq, prox...OK [1263500 terms; 601715072 terms/docs pairs; 
> 1513780631 tokens]
>     test: stored fields.......OK [8063151 total field count; avg 5.998 fields 
> per doc]
>     test: term vectors........OK [1344278 total vector count; avg 1 term/freq 
> vector fields per doc]
> No problems were detected with this index.
>
> 2011-04-06
> ________________________________
> 袁武 [GMail]
> ________________________________
> 发件人: Michael McCandless
> 发送时间: 2011-04-02  21:11:05
> 收件人: 袁武 [GMail]
> 抄送: java-user
> 主题: Re: Re: Re: A likely bug of TermsPosition.nextPosition
> So your test case still hits the exception in 3.1?
> If you fully rebuild you index in 3.1, does the exception still occur?
> Is there any way I could get access to this index?
> Do other terms besides "\1" have the problem?
> I just committed a change to the 3.x branch
> (https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x) --
> are you able to checkout this branch, build it, and run its CheckIndex
> on your index?  I'm also attaching the patch... and I could also send
> you the patched Lucene core JAR if you want.
> I'd like to see if that change to CheckIndex detects the problem with
> your index.
> Mike
> http://blog.mikemccandless.com
> 2011/4/2 袁武 [GMail] <yuanwu.m...@gmail.com>:
>> Dear Mike:
>>
>> The index was constructed using Lucene 2.9. After the problem occured, I
>> switched to recently release  Lucene 3.1, leaving the index untouched.
>>
>> Thanks to your help
>>
>> 2011-04-02
>> ________________________________
>> 袁武 [GMail]
>> ________________________________
>> 发件人: Michael McCandless
>> 发送时间: 2011-04-02  00:13:35
>> 收件人: 袁武 [GMail]
>> 抄送: java-user
>> 主题: Re: Re: A likely bug of TermsPosition.nextPosition
>> Hmm so it's not index corruption.  Curious.
>> Which Lucene version are you using?  Looks like it's 2.9, but not
>> 2.9.4?  Can you try 2.9.4 and see if you still hit the problem?
>> Can you post a small test case showing the problem, on your index?
>> Mike
>> http://blog.mikemccandless.com
>> 2011/4/1 袁武 [GMail] <yuanwu.m...@gmail.com>:
>>> Hi, Dear Mike:
>>>
>>> belows list the report of checkIndex. OS is Fedora Linux.
>>>
>>> [oracle@server bin]$ java -classpath ./ org.apache.lucene.index.CheckIndex 
>>> /data/Index/URL/Generic/ -fix
>>>
>>> NOTE: testing will be more thorough if you run java with 
>>> '-ea:org.apache.lucene...', so assertions are enabled
>>> Opening index @ /GeoGrid/data/Index/URL/Generic/
>>> Segments file=segments_ayup numSegments=1 version=FORMAT_DIAGNOSTICS 
>>> [Lucene 2.9]
>>>   1 of 1: name=_bym0 docCount=1344278
>>>     compound=false
>>>     hasProx=true
>>>     numFiles=11
>>>     size (MB)=15,979.593
>>>     diagnostics = {optimize=true, mergeFactor=3, 
>>> os.version=2.6.27.41-170.2.117.fc10.x86_64, os=Linux, mergeDocStores=true, 
>>> lucene.version=3.0-dev, source=merge, os.arch=amd64, java.version=1.6.0_21, 
>>> java.vendor=Sun Microsystems Inc.}
>>>     no deletions
>>>     test: open reader.........OK
>>>     test: fields..............OK [11 fields]
>>>     test: field norms.........OK [2 fields]
>>>     test: terms, freq, prox...
>>> OK [1263500 terms; 601715072 terms/docs pairs; 1513780631 tokens]
>>>     test: stored fields.......OK [8063151 total field count; avg 5.998 
>>> fields per doc]
>>>     test: term vectors........OK [1344278 total vector count; avg 1 
>>> term/freq vector fields per doc]
>>> No problems were detected with this index.
>>>
>>>
>>>
>>> 2011-04-01
>>> ________________________________
>>> 袁武 [GMail]
>>> ________________________________
>>> 发件人: Michael McCandless
>>> 发送时间: 2011-04-01  17:58:08
>>> 收件人: java-user
>>> 抄送: 袁武 [GMail]
>>> 主题: Re: A likely bug of TermsPosition.nextPosition
>>> Hmm this could be from a corrupted index.
>>> What version of Lucene?  What OS/filesystem?
>>> Can you run CheckIndex and post the output?
>>> Mike
>>> http://blog.mikemccandless.com
>>> 2011/3/31 袁武 [GMail] <yuanwu.m...@gmail.com>:
>>>> Hi, dear experts:
>>>>
>>>> When IndexReader.termsPositions is used to access specific terms, the call 
>>>> to TermsPosition.nextPosition success if TermsPosition.next is used. But 
>>>> if TermsPosition.skipTo is used instead of TermsPosition.next, a 
>>>> java.lang.IllegalArgumentException will be thrown, as bellows.
>>>>
>>>>  java.lang.IllegalArgumentException: Negative position
>>>>    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)
>>>>    at 
>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)
>>>>    at 
>>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:213)
>>>>    at 
>>>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
>>>>    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:92)
>>>>    at 
>>>> org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:181)
>>>>    at 
>>>> org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTermPositions.java:75)
>>>>    at 
>>>> org.apache.lucene.index.SegmentTermPositions.skipPositions(SegmentTermPositions.java:130)
>>>>    at 
>>>> org.apache.lucene.index.SegmentTermPositions.lazySkip(SegmentTermPositions.java:168)
>>>>    at 
>>>> org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPositions.java:69)
>>>>
>>>> In my further study, I found that if docid execeed 1044278, the exception 
>>>> occurs everytime, for the small ones,  the exception never occur. BTW, the 
>>>> total number of documents is about 1344278, and are never deleted.
>>>>
>>>> Thanks.
>>>>
>>>> 2011-04-01
>>>>
>>>>
>>>>
>>>> Yuan Wu [GMail]
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to