[ 
https://issues.apache.org/jira/browse/FLINK-35737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Feng Jiajie updated FLINK-35737:
--------------------------------
    Description: 
MemoryExecutionGraphInfoStore registers a ShutdownHook upon construction and 
deregisters it within its close() method.
{code:java}
public MemoryExecutionGraphInfoStore(...) {
    ...
    this.shutdownHook = ShutdownHookUtil.addShutdownHook(this, 
getClass().getSimpleName(), LOG);
}

@Override
public void close() throws IOException {
    ...
    // Remove shutdown hook to prevent resource leaks
    ShutdownHookUtil.removeShutdownHook(shutdownHook, 
getClass().getSimpleName(), LOG);
}{code}
Currently, MiniCluster instantiates a MemoryExecutionGraphInfoStore object but 
doesn't retain a reference to it, nor does it call close() during its own 
shutdown process.
{code:java}
        final DispatcherResourceManagerComponent 
dispatcherResourceManagerComponent =
                dispatcherResourceManagerComponentFactory.create(
                        ...
                        new MemoryExecutionGraphInfoStore(),  // -> new
                        ...); {code}
This behavior leads to an accumulation of ShutdownHooks when running multiple 
Flink jobs within the same local JVM. These accumulating hooks, along with 
their associated references, contribute to a memory leak.

This patch addresses the issue by ensuring that MemoryExecutionGraphInfoStore's 
close() method is invoked during MiniCluster shutdown.

https://github.com/apache/flink/pull/25009

  was:
MemoryExecutionGraphInfoStore registers a ShutdownHook upon construction and 
deregisters it within its close() method.
{code:java}
public MemoryExecutionGraphInfoStore(...) {
    ...
    this.shutdownHook = ShutdownHookUtil.addShutdownHook(this, 
getClass().getSimpleName(), LOG);
}

@Override
public void close() throws IOException {
    ...
    // Remove shutdown hook to prevent resource leaks
    ShutdownHookUtil.removeShutdownHook(shutdownHook, 
getClass().getSimpleName(), LOG);
}{code}
Currently, MiniCluster instantiates a MemoryExecutionGraphInfoStore object but 
doesn't retain a reference to it, nor does it call close() during its own 
shutdown process.
{code:java}
        final DispatcherResourceManagerComponent 
dispatcherResourceManagerComponent =
                dispatcherResourceManagerComponentFactory.create(
                        ...
                        new MemoryExecutionGraphInfoStore(),  // -> new
                        ...); {code}
This behavior leads to an accumulation of ShutdownHooks when running multiple 
Flink jobs within the same local JVM. These accumulating hooks, along with 
their associated references, contribute to a memory leak.

This patch addresses the issue by ensuring that MemoryExecutionGraphInfoStore's 
close() method is invoked during MiniCluster shutdown.


> Prevent Memory Leak by Closing MemoryExecutionGraphInfoStore on MiniCluster 
> Shutdown
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-35737
>                 URL: https://issues.apache.org/jira/browse/FLINK-35737
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Task
>    Affects Versions: 1.18.1
>            Reporter: Feng Jiajie
>            Priority: Critical
>             Fix For: 1.18.2, 1.20.0, 1.19.2
>
>
> MemoryExecutionGraphInfoStore registers a ShutdownHook upon construction and 
> deregisters it within its close() method.
> {code:java}
> public MemoryExecutionGraphInfoStore(...) {
>     ...
>     this.shutdownHook = ShutdownHookUtil.addShutdownHook(this, 
> getClass().getSimpleName(), LOG);
> }
> @Override
> public void close() throws IOException {
>     ...
>     // Remove shutdown hook to prevent resource leaks
>     ShutdownHookUtil.removeShutdownHook(shutdownHook, 
> getClass().getSimpleName(), LOG);
> }{code}
> Currently, MiniCluster instantiates a MemoryExecutionGraphInfoStore object 
> but doesn't retain a reference to it, nor does it call close() during its own 
> shutdown process.
> {code:java}
>         final DispatcherResourceManagerComponent 
> dispatcherResourceManagerComponent =
>                 dispatcherResourceManagerComponentFactory.create(
>                         ...
>                         new MemoryExecutionGraphInfoStore(),  // -> new
>                         ...); {code}
> This behavior leads to an accumulation of ShutdownHooks when running multiple 
> Flink jobs within the same local JVM. These accumulating hooks, along with 
> their associated references, contribute to a memory leak.
> This patch addresses the issue by ensuring that 
> MemoryExecutionGraphInfoStore's close() method is invoked during MiniCluster 
> shutdown.
> https://github.com/apache/flink/pull/25009



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to