[
https://issues.apache.org/jira/browse/LUCENE-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993462#comment-13993462
]
Uwe Schindler commented on LUCENE-5658:
---------------------------------------
bq. At the end of processing, if I call IndexWriter.forceMerge(1) I'll get this
error about three in five times. If I remove it, maybe one in five. I'll report
back with what happens running with SimpleFSDirectory and with a local file
system rather than GPFS.
As I analyzed already: It is more likely that the bug is caused by IBM J9. If
you change to NIOFSDirectory or SimpleFSDirectory you are just hiding the bug,
because you don't use the code anymore. We are testing Lucene with many JVMs in
our nightly builds, and unfortunately IBM J9 is causing crazy bugs from time to
time, because it miscompiles code in its JVM optimizer (AOT or how it is
called). It looks like the contents of fields in the bytebuffers and the
directory get out of sync.
So most important to me would be: Does it happen with Oracle Java 7u55?
If the bug would be caused by the filesystem, the problem would look different:
The mmap would fail ealier (see above).
> IllegalArgumentException from ByteBufferIndexInput.buildSlice
> -------------------------------------------------------------
>
> Key: LUCENE-5658
> URL: https://issues.apache.org/jira/browse/LUCENE-5658
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 4.8
> Reporter: Shai Erera
>
> I've received an email with the following stacktrace:
> {noformat}
> Exception in thread "Lucene Merge Thread #73"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.IllegalArgumentException
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:278)
> at
> org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:259)
> at
> org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:230)
> at
> org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:187)
> at
> org.apache.lucene.store.MMapDirectory$1.openFullSlice(MMapDirectory.java:211)
> at
> org.apache.lucene.store.CompoundFileDirectory.readEntries(CompoundFileDirectory.java:138)
> at
> org.apache.lucene.store.CompoundFileDirectory.<init>(CompoundFileDirectory.java:105)
> at
> org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:209)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:99)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:142)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:624)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4068)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3728)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
> {noformat}
> According to the email, it happens randomly while indexing Wikipedia, on
> 4.8.0. As far as I understood, the indexer creates 4 indexes in parallel, by
> a total of 48 threads. Each index is created in a separate directory, and
> there's no sharing of MP or MS instances between the writers (in fact,
> default settings are used). This could explain the {{Lucene Merge Thread
> #73}}. The indexing process ends w/ a {{forceMerge(1)}}. If that call is
> omitted, the exception doesn't reproduce. Also, since it doesn't happen
> always, there's no simple testcase which reproduces.
> I've asked the reporter to add more info to the issue, but opening the issue
> now so we could check and hopefully fix before 4.8.1.
> I checked the stacktrace against trunk, but not all the lines align (e.g.
> {{at
> org.apache.lucene.store.MMapDirectory$1.openFullSlice(MMapDirectory.java:211)}}
> is only in 4.8), but I haven't dug deeper yet...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
