[
https://issues.apache.org/jira/browse/LUCENE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rodrigo Trujillo updated LUCENE-4999:
-------------------------------------
Description:
When I ran the Lucene (4.2.1/4.3) test suite with IBM Java I get the following
error:
[junit4:junit4] Suite:
org.apache.lucene.search.postingshighlight.TestPostingsHighlighter
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestPostingsHighlighter -Dtests.method=testCambridgeMA
-Dtests.seed=571E16AEAF72C9F9 -Dtests.s
low=true -Dtests.locale=mt_MT -Dtests.timezone=Pacific/Kiritimati
-Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.71s J2 | TestPostingsHighlighter.testCambridgeMA <<<
[junit4:junit4] > Throwable #1: java.lang.ArrayIndexOutOfBoundsException:
Array index out of range: 37
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([571E16AEAF72C9F9:D60B7505C1DC91F8]:0)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.Passage.addMatch(Passage.java:53)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightDoc(PostingsHighlighter.java:547)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightField(PostingsHighlighter.java:425)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:364)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:268)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlight(PostingsHighlighter.java:198)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testCambridgeMA(TestPostingsHighlighter.java:373)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:738)
[junit4:junit4] 2> NOTE: test params are:
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FA
ST_DECOMPRESSION, chunkSize=386),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=386)), sim=RandomSimilarityProv
ider(queryNorm=false,coord=yes): {body=DFR I(n)Z(0.3), title=DFR I(F)Z(0.3),
id=DFR I(n)2}, locale=mt_MT, timezone=Pacific/Kiritimati
[junit4:junit4] 2> NOTE: Linux 2.6.32-279.el6.x86_64 amd64/IBM Corporation
1.6.0 (64-bit)/cpus=4,threads=1,free=10783032,total=24030208
[junit4:junit4] 2> NOTE: All tests run in this JVM: [FieldQueryTest,
FieldPhraseListTest, SimpleFragListBuilderTest, FieldTermStackTest,
OffsetLimitTokenFil
terTest, TokenSourcesTest, TestPostingsHighlighter]
[junit4:junit4] Completed on J2 in 2.46s, 23 tests, 1 error <<< FAILURES!
This error is not seen with Oracle Java.
A Google search showed that this error has already occurred in community builds
and the solution proposed was disable the IBM Java in the community tests.
I took a look in the code and found that the root of the problem is due to the
assignment of the variable "referenceSize" in RamUsageEstimator.java:
// get object reference size by getting scale factor of Object[] arrays:
try {
final Method arrayIndexScaleM = unsafeClass.getMethod("arrayIndexScale",
Class.class);
referenceSize = ((Number) arrayIndexScaleM.invoke(theUnsafe,
Object[].class)).intValue();
supportedFeatures.add(JvmFeature.OBJECT_REFERENCE_SIZE);
} catch (Exception e) {
// ignore.
}
The Java Object reference size for arrays have 8 bytes in 64-bit machines
(Oracle or IBM) and can be reduced to 4 bytes (like 32-bit JVMs) using
Compressed References and Compressed Ordinary Object Pointers (OOPs).
This options seems to be enabled by default in Oracle Java when the heap size
is under 32GB, but is not in IBM Java.
As workaround, when testing with IBM JVM I can pass the options
"-Xcompressedrefs" or "-XX:+UseCompressedOops" to Junit.
Similarly, you can reproduce the error if you pass the option
"-XX:-UseCompressedOops" when testing with Oracle Java.
The bug is in oversize method of ArrayUtil.java. It does nothing when the
object reference size (bytesPerElement) is 8.
was:
When I ran the Lucene (4.2.1/4.3) test suite with IBM Java I get the following
error:
[junit4:junit4] Suite:
org.apache.lucene.search.postingshighlight.TestPostingsHighlighter
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestPostingsHighlighter -Dtests.method=testCambridgeMA
-Dtests.seed=571E16AEAF72C9F9 -Dtests.s
low=true -Dtests.locale=mt_MT -Dtests.timezone=Pacific/Kiritimati
-Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.71s J2 | TestPostingsHighlighter.testCambridgeMA <<<
[junit4:junit4] > Throwable #1: java.lang.ArrayIndexOutOfBoundsException:
Array index out of range: 37
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([571E16AEAF72C9F9:D60B7505C1DC91F8]:0)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.Passage.addMatch(Passage.java:53)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightDoc(PostingsHighlighter.java:547)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightField(PostingsHighlighter.java:425)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:364)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:268)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlight(PostingsHighlighter.java:198)
[junit4:junit4] > at
org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testCambridgeMA(TestPostingsHighlighter.java:373)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:738)
[junit4:junit4] 2> NOTE: test params are:
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FA
ST_DECOMPRESSION, chunkSize=386),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=386)), sim=RandomSimilarityProv
ider(queryNorm=false,coord=yes): {body=DFR I(n)Z(0.3), title=DFR I(F)Z(0.3),
id=DFR I(n)2}, locale=mt_MT, timezone=Pacific/Kiritimati
[junit4:junit4] 2> NOTE: Linux 2.6.32-279.el6.x86_64 amd64/IBM Corporation
1.6.0 (64-bit)/cpus=4,threads=1,free=10783032,total=24030208
[junit4:junit4] 2> NOTE: All tests run in this JVM: [FieldQueryTest,
FieldPhraseListTest, SimpleFragListBuilderTest, FieldTermStackTest,
OffsetLimitTokenFil
terTest, TokenSourcesTest, TestPostingsHighlighter]
[junit4:junit4] Completed on J2 in 2.46s, 23 tests, 1 error <<< FAILURES!
This error is not seem with Oracle Java.
A Google search showed that this error has already occurred in community builds
and the solution proposed was disable the IBM Java in the community tests.
I took a look in the code and found that the root of the problem is due to the
assignment of the variable "referenceSize" in RamUsageEstimator.java:
// get object reference size by getting scale factor of Object[] arrays:
try {
final Method arrayIndexScaleM = unsafeClass.getMethod("arrayIndexScale",
Class.class);
referenceSize = ((Number) arrayIndexScaleM.invoke(theUnsafe,
Object[].class)).intValue();
supportedFeatures.add(JvmFeature.OBJECT_REFERENCE_SIZE);
} catch (Exception e) {
// ignore.
}
The Java Object reference size for arrays have 8 bytes in 64-bit machines
(Oracle or IBM) and can be reduced to 4 bytes (like 32-bit JVMs) using
Compressed References and Compressed Ordinary Object Pointers (OOPs).
This options seems to be enabled by default in Oracle Java when the heap size
is under 32GB, but is not in IBM Java.
As workaround, when testing with IBM JVM I can pass the options
"-Xcompressedrefs" or "-XX:+UseCompressedOops" to Junit.
Similarly, you can reproduce the error if you pass the option
"-XX:-UseCompressedOops" when testing with Oracle Java.
The bug is in oversize method of ArrayUtil.java. It does nothing when the
object reference size (bytesPerElement) is 8.
> Lucene test (testCambridgeMA) fails when JVM 64-bit does not use memory
> compression
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-4999
> URL: https://issues.apache.org/jira/browse/LUCENE-4999
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 4.3, 4.2.1
> Environment: Red Hat 6.3
> IBM Java 6 - SR13
> Ant 1.9.0
> Reporter: Rodrigo Trujillo
> Priority: Critical
>
> When I ran the Lucene (4.2.1/4.3) test suite with IBM Java I get the
> following error:
> [junit4:junit4] Suite:
> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter
> [junit4:junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestPostingsHighlighter -Dtests.method=testCambridgeMA
> -Dtests.seed=571E16AEAF72C9F9 -Dtests.s
> low=true -Dtests.locale=mt_MT -Dtests.timezone=Pacific/Kiritimati
> -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR 0.71s J2 | TestPostingsHighlighter.testCambridgeMA <<<
> [junit4:junit4] > Throwable #1: java.lang.ArrayIndexOutOfBoundsException:
> Array index out of range: 37
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([571E16AEAF72C9F9:D60B7505C1DC91F8]:0)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.Passage.addMatch(Passage.java:53)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightDoc(PostingsHighlighter.java:547)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightField(PostingsHighlighter.java:425)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:364)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:268)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlight(PostingsHighlighter.java:198)
> [junit4:junit4] > at
> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testCambridgeMA(TestPostingsHighlighter.java:373)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:738)
> [junit4:junit4] 2> NOTE: test params are:
> codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FA
> ST_DECOMPRESSION, chunkSize=386),
> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
> chunkSize=386)), sim=RandomSimilarityProv
> ider(queryNorm=false,coord=yes): {body=DFR I(n)Z(0.3), title=DFR I(F)Z(0.3),
> id=DFR I(n)2}, locale=mt_MT, timezone=Pacific/Kiritimati
> [junit4:junit4] 2> NOTE: Linux 2.6.32-279.el6.x86_64 amd64/IBM Corporation
> 1.6.0 (64-bit)/cpus=4,threads=1,free=10783032,total=24030208
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [FieldQueryTest,
> FieldPhraseListTest, SimpleFragListBuilderTest, FieldTermStackTest,
> OffsetLimitTokenFil
> terTest, TokenSourcesTest, TestPostingsHighlighter]
> [junit4:junit4] Completed on J2 in 2.46s, 23 tests, 1 error <<< FAILURES!
> This error is not seen with Oracle Java.
> A Google search showed that this error has already occurred in community
> builds and the solution proposed was disable the IBM Java in the community
> tests.
> I took a look in the code and found that the root of the problem is due to
> the assignment of the variable "referenceSize" in RamUsageEstimator.java:
> // get object reference size by getting scale factor of Object[] arrays:
> try {
> final Method arrayIndexScaleM =
> unsafeClass.getMethod("arrayIndexScale", Class.class);
> referenceSize = ((Number) arrayIndexScaleM.invoke(theUnsafe,
> Object[].class)).intValue();
> supportedFeatures.add(JvmFeature.OBJECT_REFERENCE_SIZE);
> } catch (Exception e) {
> // ignore.
> }
> The Java Object reference size for arrays have 8 bytes in 64-bit machines
> (Oracle or IBM) and can be reduced to 4 bytes (like 32-bit JVMs) using
> Compressed References and Compressed Ordinary Object Pointers (OOPs).
> This options seems to be enabled by default in Oracle Java when the heap size
> is under 32GB, but is not in IBM Java.
> As workaround, when testing with IBM JVM I can pass the options
> "-Xcompressedrefs" or "-XX:+UseCompressedOops" to Junit.
> Similarly, you can reproduce the error if you pass the option
> "-XX:-UseCompressedOops" when testing with Oracle Java.
> The bug is in oversize method of ArrayUtil.java. It does nothing when the
> object reference size (bytesPerElement) is 8.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
