[
https://issues.apache.org/jira/browse/ARROW-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419738#comment-15419738
]
Jihoon Son edited comment on ARROW-260 at 8/13/16 12:52 AM:
------------------------------------------------------------
[~julienledem], thanks for creating this ticket.
The simplest solution is to add a surefire option
-Darrow.vector.max_allocation_bytes=1048576 to limit the max allocation size.
This will affect to all unit testings, but I think this is a valid solution
because we can avoid the allocation of large memory for a vector which can
cause OOM in poor test environments.
Any thoughts on this solution?
was (Author: jihoonson):
[~julienledem], thanks for creating this ticket.
The simplest solution is to add a surefire option
-Darrow.vector.max_allocation_bytes=1048576 to limit the max allocation size.
However, this will affect to all unit testings.
Any thoughts on this solution?
> TestValueVector.testFixedVectorReallocation and
> testVariableVectorReallocation are flaky
> ----------------------------------------------------------------------------------------
>
> Key: ARROW-260
> URL: https://issues.apache.org/jira/browse/ARROW-260
> Project: Apache Arrow
> Issue Type: Test
> Components: Java - Vectors
> Reporter: Julien Le Dem
>
> The Travis-ci build has failled several times on these tests.
> It looks like they often throw OOME.
> stacktrace bellow:
> {noformat}
> testFixedVectorReallocation(org.apache.arrow.vector.TestValueVector) Time
> elapsed: 0.174 sec <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected<org.apache.arrow.vector.util.OversizedAllocationException> but
> was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at
> io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at
> org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at
> org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at
> org.apache.arrow.vector.UInt4Vector.allocateBytes(UInt4Vector.java:189)
> at org.apache.arrow.vector.UInt4Vector.allocateNew(UInt4Vector.java:171)
> at
> org.apache.arrow.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:106)
> testVariableVectorReallocation(org.apache.arrow.vector.TestValueVector) Time
> elapsed: 0.148 sec <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected<org.apache.arrow.vector.util.OversizedAllocationException> but
> was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at
> io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at
> io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at
> org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at
> org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at
> org.apache.arrow.vector.VarCharVector.allocateNew(VarCharVector.java:364)
> at
> org.apache.arrow.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:163)
> Results :
> Tests in error:
> TestValueVector.testFixedVectorReallocation » Unexpected exception,
> expected<...
> TestValueVector.testVariableVectorReallocation » Unexpected exception,
> expect...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
