Hi Timur,
Indeed, if you use JNI libraries then the memory will be off-heap and
the -XmX limit will not be respected. Currently, we don't expect users
to use JNI memory allocation. We might want to enforce a more strict
direct memory limit in the future. In this case, you would get an
OutOfMemoryE
Great answer, thanks you Max for a very detailed explanation! Illuminating
how off-heap parameter affects the memory allocation.
I read this post:
https://blogs.oracle.com/jrockit/entry/why_is_my_jvm_process_larger_t
and the thing that jumped on me is the allocation of memory for jni libs. I
do u
Hi Timur,
Shedding some light on the memory calculation:
You have a total memory size of 2500 MB for each TaskManager. The
default for 'taskmanager.memory.fraction' is 0.7. This is the fraction
of the memory used by the memory manager. When you have turned on
off-heap memory, this memory is alloc
Hello Maximilian,
I'm using 1.0.0 compiled with Scala 2.11 and Hadoop 2.7. I'm running this
on EMR. I didn't see any exceptions in other logs. What are the logs you
are interested in?
Thanks,
Timur
On Mon, Apr 25, 2016 at 3:44 AM, Maximilian Michels wrote:
> Hi Timur,
>
> Which version of Flin
Hi Timur,
Which version of Flink are you using? Could you share the entire logs?
Thanks,
Max
On Mon, Apr 25, 2016 at 12:05 PM, Robert Metzger wrote:
> Hi Timur,
>
> The reason why we only allocate 570mb for the heap is because you are
> allocating most of the memory as off heap (direct byte buf
Hi Timur,
The reason why we only allocate 570mb for the heap is because you are
allocating most of the memory as off heap (direct byte buffers).
In theory, the memory footprint of the JVM is limited to 570 (heap) + 1900
(direct mem) = 2470 MB (which is below 2500). But in practice thje JVM is
all
Hello,
Next issue in a string of things I'm solving is that my application fails
with the message 'Connection unexpectedly closed by remote task manager'.
Yarn log shows the following:
Container [pid=4102,containerID=container_1461341357870_0004_01_15] is
running beyond physical memory limit
