Hi Nikola awesome, thank you, it worked,got the basic example to work, so can now try my solutions/code.
G On Wed, May 14, 2025 at 3:52 PM George <george...@gmail.com> wrote: > Thanks. > > I would think this should rather be done up stream in the source image, > for that matter I would have expected for this package to have been > installed already as you can't really do allot onthe flink nodes with > python without this package. > > G > > On Wed, May 14, 2025 at 3:33 PM Nikola Milutinovic < > n.milutino...@levi9.com> wrote: > >> Hmm, lemme see… >> >> >> >> Oh, yes, we also had to link Python bin: >> >> >> >> ldconfig /usr/lib >> >> ln -s /usr/bin/python3 /usr/bin/python >> >> >> >> Classic trickery. >> >> >> >> Nix, >> >> >> >> *From: *George <george...@gmail.com> >> *Date: *Wednesday, May 14, 2025 at 1:09 PM >> *To: *Nikola Milutinovic <n.milutino...@levi9.com>, user@flink.apache.org >> <user@flink.apache.org> >> *Subject: *Re: Python based User defined function on Flink 1.19.1 >> >> Hi there >> >> >> >> Got it build :) >> >> >> >> I installed python3-pip in addition to the java - headless version, then >> installed the package globally and then did the clean up. >> >> >> >> I am however getting the below now. >> >> it seems to be looking for python from the flink side and not python3 >> >> >> >> ``` >> >> flink@jobmanager:/sql/prompush$ /opt/flink/bin/flink run \ >> > -py /sql/prompush/main.py \ >> > -m jobmanager:8083 >> WARNING: Unknown module: jdk.compiler specified to --add-exports >> WARNING: Unknown module: jdk.compiler specified to --add-exports >> WARNING: Unknown module: jdk.compiler specified to --add-exports >> WARNING: Unknown module: jdk.compiler specified to --add-exports >> WARNING: Unknown module: jdk.compiler specified to --add-exports >> org.apache.flink.client.program.ProgramAbortException: >> java.io.IOException: Cannot run program "python": error=2, No such file or >> directory >> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134) >> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown >> Source) >> at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> Source) >> at java.base/java.lang.reflect.Method.invoke(Unknown Source) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356) >> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223) >> at >> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113) >> at >> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) >> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) >> at >> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) >> at >> org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) >> at java.base/java.security.AccessController.doPrivileged(Unknown Source) >> at java.base/javax.security.auth.Subject.doAs(Unknown Source) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) >> at >> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >> at >> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) >> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) >> Caused by: java.io.IOException: Cannot run program "python": error=2, No >> such file or directory >> at java.base/java.lang.ProcessBuilder.start(Unknown Source) >> at java.base/java.lang.ProcessBuilder.start(Unknown Source) >> at >> org.apache.flink.client.python.PythonEnvUtils.startPythonProcess(PythonEnvUtils.java:378) >> at >> org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:492) >> at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) >> ... 17 more >> Caused by: java.io.IOException: error=2, No such file or directory >> at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) >> at java.base/java.lang.ProcessImpl.<init>(Unknown Source) >> at java.base/java.lang.ProcessImpl.start(Unknown Source) >> ... 22 more >> >> ``` >> >> >> >> On Wed, May 14, 2025 at 11:50 AM George <george...@gmail.com> wrote: >> >> Hi there >> >> >> >> Wonder if you can assist. >> >> >> >> First thinking of installing apache-flink into a dedicated environment, >> see my docker file attached... this does result in the attached err.txt >> stack though. >> >> Wonder if it does not simply make more sense to install apache-flink >> globally, then everyone that works with python on this node would have >> access to it. >> >> >> >> personally i would have expected this have formed part of this image >> available... >> >> >> >> G >> >> >> >> On Wed, May 14, 2025 at 11:18 AM Nikola Milutinovic < >> n.milutino...@levi9.com> wrote: >> >> Hi George. >> >> >> >> We saw the same problem, running Apache Flink 1.19 and 1.20 images. The >> cause is that Flink image provides a JRE and you need JDK to build/install >> PyFlink. And, oddly enough, I think it was only on ARM64 images. Amd64 was >> OK, I think. So, Mac M1, M2, M3… >> >> >> >> Our Docker file for building our custom image has some extra steps, to >> install JDK, pip-install and then cleanup. >> >> >> >> RUN apt-get update -y \ >> >> && apt-get upgrade -y \ >> >> && apt-get install --no-install-recommends -y \ >> >> … >> >> openjdk-17-jdk-headless \ >> >> ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 >> >> >> >> After PIP install our requirement.txt, we do cleanup. >> >> >> >> # Cleanup JDK >> >> RUN apt-get remove -y openjdk-17-jdk-headless >> >> ENV JAVA_HOME=/opt/java/openjdk >> >> >> >> Hope it helps. >> >> >> >> Nix. >> >> >> >> *From: *George <george...@gmail.com> >> *Date: *Wednesday, May 14, 2025 at 10:23 AM >> *To: *user@flink.apache.org <user@flink.apache.org> >> *Subject: *Python based User defined function on Flink 1.19.1 >> >> Hi all >> >> >> >> My Flink Jobmanager and Taskmanager is based on the community 1.19.1 on >> the Java 17 image. >> >> >> >> I've extended this by installing Python 3.11 and python3.11-venv. >> >> >> >> created my venv environment, activated it, >> >> when trying to pip install apache-flink. I first got an error saying >> /opt/java/openjdk/include does not exist... so created it. >> >> Next attempt ended in the below error. >> >> >> >> Please advise. >> >> >> >> G >> >> >> >> ``` >> >> venv) flink@jobmanager:/sql/prompush$ pip install apache-flink >> Collecting apache-flink >> Downloading apache-flink-2.0.0.tar.gz (1.5 MB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 1.9 MB/s eta >> 0:00:00 >> Installing build dependencies ... done >> Getting requirements to build wheel ... done >> Preparing metadata (pyproject.toml) ... done >> Collecting python-dateutil<3,>=2.8.0 >> Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) >> Collecting requests>=2.26.0 >> Using cached requests-2.32.3-py3-none-any.whl (64 kB) >> Collecting pyarrow>=5.0.0 >> Downloading pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl (40.7 >> MB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.7/40.7 MB 415.8 kB/s >> eta 0:00:00 >> Collecting pytz>=2018.3 >> Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB) >> Collecting fastavro!=1.8.0,>=1.1.0 >> Using cached >> fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl >> (3.3 MB) >> Collecting pemja==0.4.1 >> Downloading pemja-0.4.1.tar.gz (50 kB) >> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 KB 600.0 kB/s >> eta 0:00:00 >> Installing build dependencies ... done >> Getting requirements to build wheel ... error >> error: subprocess-exited-with-error >> >> × Getting requirements to build wheel did not run successfully. >> │ exit code: 255 >> ╰─> [1 lines of output] >> jni.h should be in '/opt/java/openjdk/include/jni.h' but doesn't >> exist. Please check you've installed the JDK properly. >> [end of output] >> >> note: This error originates from a subprocess, and is likely not a >> problem with pip. >> error: subprocess-exited-with-error >> >> × Getting requirements to build wheel did not run successfully. >> │ exit code: 255 >> ╰─> See above for output. >> >> note: This error originates from a subprocess, and is likely not a >> problem with pip. >> >> ``` >> >> >> >> -- >> >> You have the obligation to inform one honestly of the risk, and as a >> person >> you are committed to educate yourself to the total risk in any activity! >> >> Once informed & totally aware of the risk, >> every fool has the right to kill or injure themselves as they see fit! >> >> >> >> >> -- >> >> You have the obligation to inform one honestly of the risk, and as a >> person >> you are committed to educate yourself to the total risk in any activity! >> >> Once informed & totally aware of the risk, >> every fool has the right to kill or injure themselves as they see fit! >> >> >> >> >> -- >> >> You have the obligation to inform one honestly of the risk, and as a >> person >> you are committed to educate yourself to the total risk in any activity! >> >> Once informed & totally aware of the risk, >> every fool has the right to kill or injure themselves as they see fit! >> > > > -- > You have the obligation to inform one honestly of the risk, and as a person > you are committed to educate yourself to the total risk in any activity! > > Once informed & totally aware of the risk, > every fool has the right to kill or injure themselves as they see fit! > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
