Hi there Got it build :)
I installed python3-pip in addition to the java - headless version, then installed the package globally and then did the clean up. I am however getting the below now. it seems to be looking for python from the flink side and not python3 ``` flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \ > -py /sql/prompush/main.py \ > -m jobmanager:8083 WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports org.apache.flink.client.program.ProgramAbortException: java.io.IOException: Cannot run program "python": error=2, No such file or directory at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) at java.base/java.security.AccessController.doPrivileged(Unknown Source) at java.base/javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) Caused by: java.io.IOException: Cannot run program "python": error=2, No such file or directory at java.base/java.lang.ProcessBuilder.start(Unknown Source) at java.base/java.lang.ProcessBuilder.start(Unknown Source) at org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378) at org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492) at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) ... 17 more Caused by: java.io.IOException: error=2, No such file or directory at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) at java.base/java.lang.ProcessImpl.<init>(Unknown Source) at java.base/java.lang.ProcessImpl.start(Unknown Source) ... 22 more ``` On Wed, May 14, 2025 at 11:50 AM George <george...@gmail.com> wrote: > Hi there > > Wonder if you can assist. > > First thinking of installing apache-flink into a dedicated environment, > see my docker file attached... this does result in the attached err.txt > stack though. > Wonder if it does not simply make more sense to install apache-flink > globally, then everyone that works with python on this node would have > access to it. > > personally i would have expected this have formed part of this image > available... > > G > > On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic < > n.milutino...@levi9.com> wrote: > >> Hi George. >> >> >> >> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The >> cause is that Flink image provides a JRE and you need JDK to build/install >> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was >> OK, I think. So, Mac M1, M2, M3… >> >> >> >> Our Docker file for building our custom image has some extra steps, to >> install JDK, pip-install and then cleanup. >> >> >> >> RUN apt-get update -y \ >> >> && apt-get upgrade -y \ >> >> && apt-get install --no-install-recommends -y \ >> >> … >> >> openjdk-17-jdk-headless \ >> >> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 >> >> >> >> After PIP install our requirement.txt, we do cleanup. >> >> >> >> # Cleanup JDK >> >> RUN apt-get remove -y openjdk-17-jdk-headless >> >> ENV JAVA_HOME=/opt/java/openjdk >> >> >> >> Hope it helps. >> >> >> >> Nix. >> >> >> >> *From: *George <george...@gmail.com> >> *Date: *Wednesday, May 14, 2025 at 10:23 AM >> *To: *user@flink.apache.org <user@flink.apache.org> >> *Subject: *Python based User defined function on Flink 1.19.1 >> >> Hi all >> >> >> >> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on >> the Java 17 image. >> >> >> >> I've extended this by installing Python 3.11 and python3.11-venv. >> >> >> >> created my venv environment, activated it, >> >> when trying to pip install apache-flink. I first got an error saying >> /opt/java/openjdk/include does not exist... so created it. >> >> Next attempt ended in the below error. >> >> >> >> Please advise. >> >> >> >> G >> >> >> >> ``` >> >> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink >> Collecting apache-flink >> Downloading apache-flink-2.0.0.tar.gz (1.5 MB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta >> 0:00:00 >> Installing build dependencies ... done >> Getting requirements to build wheel ... done >> Preparing metadata (pyproject.toml) ... done >> Collecting python-dateutil<3,>=2.8.0 >> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) >> Collecting requests>=2.26.0 >> Using cached requests-2.32.3-py3-none-any.whl (64 kB) >> Collecting pyarrow>=5.0.0 >> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 >> MB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s >> eta 0:00:00 >> Collecting pytz>=2018.3 >> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB) >> Collecting fastavro!=1.8.0,>=1.1.0 >> Using cached >> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl >> (3.3 MB) >> Collecting pemja==0.4.1 >> Downloading pemja-0.4.1.tar.gz (50 kB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s >> eta 0:00:00 >> Installing build dependencies ... done >> Getting requirements to build wheel ... error >> error: subprocess-exited-with-error >> >> × Getting requirements to build wheel did not run successfully. >> │ exit code: 255 >> ╰─> [1 lines of output] >> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't >> exist. Please check you've installed the JDK properly. >> [end of output] >> >> note: This error originates from a subprocess, and is likely not a >> problem with pip. >> error: subprocess-exited-with-error >> >> × Getting requirements to build wheel did not run successfully. >> │ exit code: 255 >> ╰─> See above for output. >> >> note: This error originates from a subprocess, and is likely not a >> problem with pip. >> >> ``` >> >> >> >> -- >> >> You have the obligation to inform one honestly of the risk, and as a >> person >> you are committed to educate yourself to the total risk in any activity! >> >> Once informed & totally aware of the risk, >> every fool has the right to kill or injure themselves as they see fit! >> > > > -- > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
