Hmm, lemme see… Oh, yes, we also had to link Python bin:
ldconfig /usr/lib ln -s /usr/bin/python3 /usr/bin/python Classic trickery. Nix, From: George <george...@gmail.com> Date: Wednesday, May 14, 2025 at 1:09 PM To: Nikola Milutinovic <n.milutino...@levi9.com>, user@flink.apache.org <user@flink.apache.org> Subject: Re: Python based User defined function on Flink 1.19.1 Hi there Got it build :) I installed python3-pip in addition to the java - headless version, then installed the package globally and then did the clean up. I am however getting the below now. it seems to be looking for python from the flink side and not python3 ``` flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \ > -py /sql/prompush/main.py \ > -m jobmanager:8083 WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports org.apache.flink.client.program.ProgramAbortException: java.io.IOException: Cannot run program "python": error=2, No such file or directory at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at java.base/javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) Caused by: java.io.IOException: Cannot run program "python": error=2, No such file or directory at java.base/java.lang.ProcessBuilder.start(Unknown Source) at java.base/java.lang.ProcessBuilder.start(Unknown Source) at org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378) at org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492) at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) ... 17 more Caused by: java.io.IOException: error=2, No such file or directory at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) at java.base/java.lang.ProcessImpl.<init>(Unknown Source) at java.base/java.lang.ProcessImpl.start(Unknown Source) ... 22 more ``` On Wed, May 14, 2025 at 11:50 AM George <george...@gmail.com<mailto:george...@gmail.com>> wrote: Hi there Wonder if you can assist. First thinking of installing apache-flink into a dedicated environment, see my docker file attached... this does result in the attached err.txt stack though. Wonder if it does not simply make more sense to install apache-flink globally, then everyone that works with python on this node would have access to it. personally i would have expected this have formed part of this image available... G On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic <n.milutino...@levi9.com<mailto:n.milutino...@levi9.com>> wrote: Hi George. We saw the same problem, running Apache Flink 1.19 and 1.20 images. The cause is that Flink image provides a JRE and you need JDK to build/install PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was OK, I think. So, Mac M1, M2, M3… Our Docker file for building our custom image has some extra steps, to install JDK, pip-install and then cleanup. RUN apt-get update -y \ && apt-get upgrade -y \ && apt-get install --no-install-recommends -y \ … openjdk-17-jdk-headless \ ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 After PIP install our requirement.txt, we do cleanup. # Cleanup JDK RUN apt-get remove -y openjdk-17-jdk-headless ENV JAVA_HOME=/opt/java/openjdk Hope it helps. Nix. From: George <george...@gmail.com<mailto:george...@gmail.com>> Date: Wednesday, May 14, 2025 at 10:23 AM To: user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Python based User defined function on Flink 1.19.1 Hi all My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on the Java 17 image. I've extended this by installing Python 3.11 and python3.11-venv. created my venv environment, activated it, when trying to pip install apache-flink. I first got an error saying /opt/java/openjdk/include does not exist... so created it. Next attempt ended in the below error. Please advise. G ``` venv) flink@jobmanager:/sql/prompush$ pip install apache-flink Collecting apache-flink Downloading apache-flink-2.0.0.tar.gz (1.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting python-dateutil<3,>=2.8.0 Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting requests>=2.26.0 Using cached requests-2.32.3-py3-none-any.whl (64 kB) Collecting pyarrow>=5.0.0 Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta 0:00:00 Collecting pytz>=2018.3 Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB) Collecting fastavro!=1.8.0,>=1.1.0 Using cached fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB) Collecting pemja==0.4.1 Downloading pemja-0.4.1.tar.gz (50 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 255 ╰─> [1 lines of output] jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't exist. Please check you've installed the JDK properly. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 255 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. ``` -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit! -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit! -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
