Thanks. I would think this should rather be done up stream in the source image, for that matter I would have expected for this package to have been installed already as you can't really do allot onthe flink nodes with python without this package.
G On Wed, May 14, 2025 at 3:33 PM Nikola Milutinovic <n.milutino...@levi9.com> wrote: > Hmm, lemme see… > > > > Oh, yes, we also had to link Python bin: > > > > ldconfig /usr/lib > > ln -s /usr/bin/python3 /usr/bin/python > > > > Classic trickery. > > > > Nix, > > > > *From: *George <george...@gmail.com> > *Date: *Wednesday, May 14, 2025 at 1:09 PM > *To: *Nikola Milutinovic <n.milutino...@levi9.com>, user@flink.apache.org > <user@flink.apache.org> > *Subject: *Re: Python based User defined function on Flink 1.19.1 > > Hi there > > > > Got it build :) > > > > I installed python3-pip in addition to the java - headless version, then > installed the package globally and then did the clean up. > > > > I am however getting the below now. > > it seems to be looking for python from the flink side and not python3 > > > > ``` > > flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \ > > -py /sql/prompush/main.py \ > > -m jobmanager:8083 > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > org.apache.flink.client.program.ProgramAbortException: > java.io.IOException: Cannot run program "python": error=2, No such file or > directory > at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) > at > org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) > at > org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) > at java.base/java.security.AccessController.doPrivileged(Unknown Source) > at java.base/javax.security.auth.Subject.doAs(Unknown Source) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) > Caused by: java.io.IOException: Cannot run program "python": error=2, No > such file or directory > at java.base/java.lang.ProcessBuilder.start(Unknown Source) > at java.base/java.lang.ProcessBuilder.start(Unknown Source) > at > org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378) > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492) > at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ... 17 more > Caused by: java.io.IOException: error=2, No such file or directory > at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) > at java.base/java.lang.ProcessImpl.<init>(Unknown Source) > at java.base/java.lang.ProcessImpl.start(Unknown Source) > ... 22 more > > ``` > > > > On Wed, May 14, 2025 at 11:50 AM George <george...@gmail.com> wrote: > > Hi there > > > > Wonder if you can assist. > > > > First thinking of installing apache-flink into a dedicated environment, > see my docker file attached... this does result in the attached err.txt > stack though. > > Wonder if it does not simply make more sense to install apache-flink > globally, then everyone that works with python on this node would have > access to it. > > > > personally i would have expected this have formed part of this image > available... > > > > G > > > > On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic < > n.milutino...@levi9.com> wrote: > > Hi George. > > > > We saw the same problem, running Apache Flink 1.19 and 1.20 images. The > cause is that Flink image provides a JRE and you need JDK to build/install > PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was > OK, I think. So, Mac M1, M2, M3… > > > > Our Docker file for building our custom image has some extra steps, to > install JDK, pip-install and then cleanup. > > > > RUN apt-get update -y \ > > && apt-get upgrade -y \ > > && apt-get install --no-install-recommends -y \ > > … > > openjdk-17-jdk-headless \ > > ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 > > > > After PIP install our requirement.txt, we do cleanup. > > > > # Cleanup JDK > > RUN apt-get remove -y openjdk-17-jdk-headless > > ENV JAVA_HOME=/opt/java/openjdk > > > > Hope it helps. > > > > Nix. > > > > *From: *George <george...@gmail.com> > *Date: *Wednesday, May 14, 2025 at 10:23 AM > *To: *user@flink.apache.org <user@flink.apache.org> > *Subject: *Python based User defined function on Flink 1.19.1 > > Hi all > > > > My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on > the Java 17 image. > > > > I've extended this by installing Python 3.11 and python3.11-venv. > > > > created my venv environment, activated it, > > when trying to pip install apache-flink. I first got an error saying > /opt/java/openjdk/include does not exist... so created it. > > Next attempt ended in the below error. > > > > Please advise. > > > > G > > > > ``` > > venv) flink@jobmanager:/sql/prompush$ pip install apache-flink > Collecting apache-flink > Downloading apache-flink-2.0.0.tar.gz (1.5 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta > 0:00:00 > Installing build dependencies ... done > Getting requirements to build wheel ... done > Preparing metadata (pyproject.toml) ... done > Collecting python-dateutil<3,>=2.8.0 > Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) > Collecting requests>=2.26.0 > Using cached requests-2.32.3-py3-none-any.whl (64 kB) > Collecting pyarrow>=5.0.0 > Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 > MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s eta > 0:00:00 > Collecting pytz>=2018.3 > Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB) > Collecting fastavro!=1.8.0,>=1.1.0 > Using cached > fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > (3.3 MB) > Collecting pemja==0.4.1 > Downloading pemja-0.4.1.tar.gz (50 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s eta > 0:00:00 > Installing build dependencies ... done > Getting requirements to build wheel ... error > error: subprocess-exited-with-error > > × Getting requirements to build wheel did not run successfully. > │ exit code: 255 > ╰─> [1 lines of output] > jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't > exist. Please check you've installed the JDK properly. > [end of output] > > note: This error originates from a subprocess, and is likely not a > problem with pip. > error: subprocess-exited-with-error > > × Getting requirements to build wheel did not run successfully. > │ exit code: 255 > ╰─> See above for output. > > note: This error originates from a subprocess, and is likely not a problem > with pip. > > ``` > > > > -- > > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! > > > > > -- > > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! > > > > > -- > > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
