[
https://issues.apache.org/jira/browse/ARROW-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419253#comment-15419253
]
Julien Le Dem commented on ARROW-260:
-------------------------------------
from jihoonson: https://github.com/apache/arrow/pull/116#issuecomment-239468610
{quote}
I'm also suffering from this error, and found the reason. The
BaseValueVector.MAX_ALLOCATION_SIZE value used in TestValueVector is declared
as a static variable which means its value is initialized when the
BaseValueVector class is loaded. Obviously, changing
BaseValueVector.MAX_ALLOCATION_SIZE by setting the
"arrow.vector.max_allocation_bytes" property in TestValueVector doesn't work.
As a result, this test tries to allocate a very large array of the size of
Integer.MAX_VALUE, which causes OOM.
I think that this test looks fragile, and thus we need to make the allocation
size small. However, reducing the allocation size causes another problem. Many
tests in TestValueVector expect the OversizedAllocationException when value
allocation reaches to MAX_VALUE_ALLOCATION. So, reducing the allocation size
makes OversizedAllocation tests difficult.
I would like to separate all these OversizedAllocation tests, but not sure this
is a right way. Please give me some help.
{quote}
> TestValueVector.testFixedVectorReallocation and
> testVariableVectorReallocation are flaky
> ----------------------------------------------------------------------------------------
>
> Key: ARROW-260
> URL: https://issues.apache.org/jira/browse/ARROW-260
> Project: Apache Arrow
> Issue Type: Test
> Components: Java - Vectors
> Reporter: Julien Le Dem
>
> The Travis-ci build has failled several times on these tests.
> It looks like they often throw OOME.
> stacktrace bellow:
> {noformat}
> testFixedVectorReallocation(org.apache.arrow.vector.TestValueVector) Time
> elapsed: 0.174 sec <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected<org.apache.arrow.vector.util.OversizedAllocationException> but
> was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at
> io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at
> org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at
> org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at
> org.apache.arrow.vector.UInt4Vector.allocateBytes(UInt4Vector.java:189)
> at org.apache.arrow.vector.UInt4Vector.allocateNew(UInt4Vector.java:171)
> at
> org.apache.arrow.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:106)
> testVariableVectorReallocation(org.apache.arrow.vector.TestValueVector) Time
> elapsed: 0.148 sec <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected<org.apache.arrow.vector.util.OversizedAllocationException> but
> was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at
> io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at
> org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at
> org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at
> org.apache.arrow.vector.VarCharVector.allocateNew(VarCharVector.java:364)
> at
> org.apache.arrow.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:163)
> Results :
> Tests in error:
> TestValueVector.testFixedVectorReallocation » Unexpected exception,
> expected<...
> TestValueVector.testVariableVectorReallocation » Unexpected exception,
> expect...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
