Bowen Li created FLINK-40628:
--------------------------------
Summary: PyFlink TTL initialization can hang task threads and
leave jobs stuck in INITIALIZING
Key: FLINK-40628
URL: https://issues.apache.org/jira/browse/FLINK-40628
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.19.3
Reporter: Bowen Li
Assignee: Bowen Li
Fix For: 2.4.0
*User impact*
Python operators can remain stuck in INITIALIZING, preventing the job from
processing data and completing checkpoints. The TaskManager process can remain
alive while its task threads are blocked, so process-liveness checks may not
detect the failure. Restarting the application can recover it, but the problem
can recur.
*Cause*
With default TTL background cleanup, embedded state initialization follows:
RuntimeContext.get_state()
→ default RocksDB cleanup strategy
→ Duration.of_days(30)
→ get_gateway()
→ subprocess.Popen(preexec_fn=...)
This unexpectedly launches a separate gateway process from the multithreaded
TaskManager. The child process can deadlock before exec, leaving the calling
task thread waiting and the job unable to finish initialization. Python
documents this deadlock risk.
*Verification*
The unexpected gateway request was reproduced through a real JVM/Pemja/RocksDB
job using a Flink 1.19.1-derived distribution, Python 3.12.9, and Pemja 0.5.5.
The same code path remains in upstream 1.19.3, 1.20.5, 2.3.0, and inspected
master.
The production initialization hang was observed; its exact blocked lock was not
captured.
*Proposed fix*
Avoid constructing a Py4J-backed Duration during embedded TTL initialization.
Keep the default duration as a Python value and convert it through Pemja. Add a
regression test ensuring RuntimeContext.get_state() with default TTL never
requests a gateway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
