Hi Till,
Thanks a lot for the prompt response, please see below information.
1. how much memory assign to JM pod?
6g for container memory limit, 5g for jobmanager.heap.size, I think this is
the only available jm memory configuration for flink 1.10.2
2. Have you tried with newer Flink versions?
I am actually using Apache Beam, so the latest version they support for
Flink is 1.10
3. What statebackend is used?
FsStateBackend, and the checkpoint size is around 12MB from checkpoint
metrics, so I think it is not get inlined
4. What is state.checkpoints.num-retained?
I did not configure this explicitly, so by default only 1 should be retained
5. Anything suspicious from JM log?
There is no Exception nor Error, the only thing I see is the below logs
keeps on repeating
{"@timestamp":"2020-10-23T16:05:20.350Z","@version":"1","message":"Disabling
threads for Delete operation as thread count 0 is <=
1","logger_name":"org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor","thread_name":"jobmanager-future-thread-4","level":"WARN","level_value":30000}
6. JVM args obtained vis jcmd
-Xms5120m -Xmx5120m -XX:MaxGCPauseMillis=20 -XX:-OmitStackTraceInFastThrow
7. Heap info returned by jcmd <pid> GC.heap_info
it suggested only about 1G of the heap is used
garbage-first heap total 5242880K, used 1123073K [0x00000006c0000000,
0x0000000800000000)
region size 2048K, 117 young (239616K), 15 survivors (30720K)
Metaspace used 108072K, capacity 110544K, committed 110720K,
reserved 1146880K
class space used 12963K, capacity 13875K, committed 13952K, reserved
1048576K
8. top -p <pid>
it suggested for flink job manager java process 4.8G of physical memory is
consumed
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 13.356g 4.802g 22676 S 6.0 7.6 37:48.62 java
Thanks a lot!
Eleanore
On Fri, Oct 23, 2020 at 4:19 AM Till Rohrmann <trohrm...@apache.org> wrote:
> Hi Eleanore,
>
> how much memory did you assign to the JM pod? Maybe the limit is so high
> that it takes a bit of time until GC is triggered. Have you tried whether
> the same problem also occurs with newer Flink versions?
>
> The difference between checkpoints enabled and disabled is that the JM
> needs to do a bit more bookkeeping in order to track the completed
> checkpoints. If you are using the HeapStateBackend, then all states smaller
> than state.backend.fs.memory-threshold will get inlined, meaning that they
> are sent to the JM and stored in the checkpoint meta file. This can
> increase the memory usage of the JM process. Depending on
> state.checkpoints.num-retained this can grow as large as number retained
> checkpoints times the checkpoint size. However, I doubt that this adds up
> to several GB of additional space.
>
> In order to better understand the problem, the debug logs of your JM could
> be helpful. Also a heap dump might be able to point us towards the
> component which is eating up so much memory.
>
> Cheers,
> Till
>
> On Thu, Oct 22, 2020 at 4:56 AM Eleanore Jin <eleanore....@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I have a flink job running version 1.10.2, it simply read from a kafka
>> topic with 96 partitions, and output to another kafka topic.
>>
>> It is running in k8s, with 1 JM (not in HA mode), 12 task managers each
>> has 4 slots.
>> The checkpoint persists the snapshot to azure blob storage, checkpoints
>> interval every 3 seconds, with 10 seconds timeout and minimum pause of 1
>> second.
>>
>> I observed that the job manager pod memory usage grows over time, any
>> hints on why this is the case? And the memory usage for JM is significantly
>> more compared to no checkpoint enabled.
>> [image: image.png]
>>
>> Thanks a lot!
>> Eleanore
>>
>
