[
https://issues.apache.org/jira/browse/FLINK-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402231#comment-15402231
]
Maximilian Michels commented on FLINK-4094:
-------------------------------------------
Thanks for elaborating on your thoughts [~ram_krish]! I was assuming that the
memory segments actually would be managed by a pool regardless of preallocation
set to false or true. You're right that without preallocation, segments are
requested and released as they are needed. This DOES NOT work for the offheap
memory segments (at least not how it is implemented now).
Two possible fixes that come to my mind:
1) Implement proper clearing of the offheap memory segments and keeping the
preallocation:false behavior: http://stackoverflow.com/a/8462690/2225100
2) Use pooling even with preallocation:false which only differs to
preallocation:true that the memory is allocated lazily over time.
I would say, let's go with 2) which should be easily solvable.
{quote}
I would rather say that it is better we do internal management of offheap
buffers. We should create a pool from which the buffers are allocated and if
the pool is of fixed size and we have requests for more buffers than the size
of the pool we should allocate them onheap only. (if that is acceptable).
{quote}
In the case of offheap managed memory, the upper bound of the memory should be
respected in the same way as for heap memory. No additional memory should be
allocated on the heap instead. The user has to increase the memory size of the
job fails.
> Off heap memory deallocation might not properly work
> ----------------------------------------------------
>
> Key: FLINK-4094
> URL: https://issues.apache.org/jira/browse/FLINK-4094
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
> Affects Versions: 1.1.0
> Reporter: Till Rohrmann
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 1.1.0
>
>
> A user reported that off-heap memory is not properly deallocated when setting
> {{taskmanager.memory.preallocate:false}} (per default) [1]. This can cause
> the TaskManager process being killed by the OS.
> It should be possible to execute multiple batch jobs with preallocation
> turned off. No longer used direct memory buffers should be properly garbage
> collected so that the JVM process does not exceed it's maximum memory bounds.
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/offheap-memory-allocation-and-memory-leak-bug-td12154.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
