Martijn Visser created FLINK-39895:
--------------------------------------
Summary: PyFlink tests fail on release-1.20: setuptools 82 removed
pkg_resources, breaking the Beam Python harness
Key: FLINK-39895
URL: https://issues.apache.org/jira/browse/FLINK-39895
Project: Flink
Issue Type: Bug
Components: Build System / CI, Tests
Affects Versions: 1.20.5
Reporter: Martijn Visser
Assignee: Martijn Visser
All PyFlink unit tests (e.g. {{pyflink/datastream/tests/test_window.py}},
{{test_stream_execution_environment.py}}) and the PyFlink end-to-end test fail
on the {{release-1.20}} branch nightly. Example:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75796
The Java side fails creating the Beam stage bundle factory:
{noformat}
java.lang.RuntimeException: Failed to create stage bundle factory! ...
Caused by: java.lang.IllegalStateException: Process died with exit code 0
{noformat}
The Beam Python harness ({{beam_boot.py}}) dies on import:
{noformat}
File ".../apache_beam/runners/portability/stager.py", line 63, in <module>
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
{noformat}
{{setuptools}} 82.0.0 removed the {{pkg_resources}} module. apache-beam 2.48.0
(the version embedded on release-1.20, pinned {{apache-beam>=2.43.0,<2.49.0}})
still imports {{pkg_resources}}. The pins in
{{flink-python/dev/dev-requirements.txt}}
and {{flink-python/pyproject.toml}} were open-ended ({{setuptools>=75.3}}), so
pip
resolved {{setuptools-82.0.1}} and the harness exits before serving the gRPC
FnHarness, hence "Process died with exit code 0".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
