[
https://issues.apache.org/jira/browse/FLINK-40628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-40628.
---------------------------
Fix Version/s: 1.20.6
2.3.1
Resolution: Fixed
Fixed in:
- master via f5aaa7361d0bb1bc1dcf7c37b10bc8773f40ad3c
- release-2.3 via 0ad10a5d150e81960ce9c94e47f07082d1084e28
- release-1.20 via 7a048417bfe41e83bb5555dbfd4cee265c7a95e3
> PyFlink TTL initialization can hang task threads and leave jobs stuck in
> INITIALIZING
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40628
> URL: https://issues.apache.org/jira/browse/FLINK-40628
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.19.3
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.6, 2.3.1, 2.4.0
>
>
> *User impact*
> Python operators can remain stuck in INITIALIZING, preventing the job from
> processing data and completing checkpoints. The TaskManager process can
> remain alive while its task threads are blocked, so process-liveness checks
> may not detect the failure. Restarting the application can recover it, but
> the problem can recur.
> *Cause*
> With default TTL background cleanup, embedded state initialization follows:
> RuntimeContext.get_state()
> → default RocksDB cleanup strategy
> → Duration.of_days(30)
> → get_gateway()
> → subprocess.Popen(preexec_fn=...)
> This unexpectedly launches a separate gateway process from the multithreaded
> TaskManager. The child process can deadlock before exec, leaving the calling
> task thread waiting and the job unable to finish initialization. Python
> documents this deadlock risk.
> *Verification*
> The unexpected gateway request was reproduced through a real
> JVM/Pemja/RocksDB job using a Flink 1.19.1-derived distribution, Python
> 3.12.9, and Pemja 0.5.5. The same code path remains in upstream 1.19.3,
> 1.20.5, 2.3.0, and inspected master.
> The production initialization hang was observed; its exact blocked lock was
> not captured.
> *Proposed fix*
> Avoid constructing a Py4J-backed Duration during embedded TTL initialization.
> Keep the default duration as a Python value and convert it through Pemja. Add
> a regression test ensuring RuntimeContext.get_state() with default TTL never
> requests a gateway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
