[
https://issues.apache.org/jira/browse/FLINK-32136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17733327#comment-17733327
]
Dian Fu edited comment on FLINK-32136 at 6/16/23 5:25 AM:
----------------------------------------------------------
[~wash] I have submitted a PR (https://github.com/apache/flink/pull/22802).
Could you help to review? It would be great if you could also help to verify it~
was (Author: dianfu):
[~wash] I have submitted a PR. Could you help to review? It would be great if
you could also help to verify it~
> Pyflink gateway server launch fails when purelib != platlib
> -----------------------------------------------------------
>
> Key: FLINK-32136
> URL: https://issues.apache.org/jira/browse/FLINK-32136
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.13.3
> Reporter: William Ashley
> Priority: Major
> Labels: pull-request-available
>
> On distros where python's {{purelib}} is different than {{platlib}} (e.g.
> Amazon Linux 2, but from my research it's all of the Redhat-based ones), you
> wind up with components of packages being installed across two different
> locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and
> {{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
> {{_find_flink_home}}
> [handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60]
> this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}}
> environment variable was the one being used. However, from 1.13.3, a
> refactoring of {{launch_gateway_server_process}}
> ([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200]
>
> [1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280])
> re-ordered some method calls. {{{}prepare_environment_variable{}}}'s
> [non-awareness|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L94C67-L95]
> of multiple homes and setting of {{FLINK_LIB_DIR}} now is the one that
> matters, and it is the incorrect location.
> I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not
> exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} ==
> {{{}purelib{}}}).
> Repro steps on Amazon Linux 2
> {quote}{{yum -y install python3 java-11}}
> {{pip3 install apache-flink==1.13.3}}
> {{python3 -c 'from pyflink.table import EnvironmentSettings ;
> EnvironmentSettings.new_instance()'}}
> {quote}
> The resulting error is
> {quote}{{The flink-python jar is not found in the opt folder of the
> FLINK_HOME: /usr/local/lib64/python3.7/site-packages/pyflink}}
> {{Error: Could not find or load main class
> org.apache.flink.client.python.PythonGatewayServer}}
> {{Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.client.python.PythonGatewayServer}}
> {{Traceback (most recent call last):}}
> {{ File "<string>", line 1, in <module>}}
> {{ File
> "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
> line 214, in new_instance}}
> {{ return EnvironmentSettings.Builder()}}
> {{ File
> "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
> line 48, in {_}{{_}}init{{_}}{_}}}
> {{ gateway = get_gateway()}}
> {{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
> line 62, in get_gateway}}
> {{ _gateway = launch_gateway()}}
> {{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
> line 112, in launch_gateway}}
> {{ raise Exception("Java gateway process exited before sending its port
> number")}}
> {{Exception: Java gateway process exited before sending its port number}}
> {quote}
> The flink home under /lib64/ does not contain the jar, but it is in the /lib/
> location
> {quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink
> -name "flink-python*.jar"}}
> {{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name
> "flink-python*.jar"}}
> {{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
