Thanks for the detailed info Xingbo, I am pretty new to the PyFlink module so was mainly trying to experiment a little :)
I think supporting it in Flink 1.16 is more than enough, it would probably be an overkill to try to backport this. The M1 issue only really occurs in local development/testing. Gyula On Fri, Jul 8, 2022 at 5:51 AM Xingbo Huang <[email protected]> wrote: > Hi Gyula, > > According to the log, we can see that you downloaded the source package of > pemja, not the wheel package of pemja[1]. I guess you are using the m1 > machine. If you install pemja from the source package, you need to have > JDK, gcc tools and CPython with Numpy in the environment. I believe this > can be solved after you prepared those tools, but other dependencies of > pyflink 1.15 do not support m1, which makes PyFlink 1.15 unable to install > and use in m1. > > We have supported m1 in release 1.16[2]. If a large number of m1 users > have big demand for PyFlink 1.15, I think we need to consider whether it is > necessary to backport this support to 1.15, which will break our > compatibility issues between minor versions. > Best, > Xingbo > > [1] https://pypi.org/project/pemja/0.1.4/ > [2] https://issues.apache.org/jira/browse/FLINK-25188 > > Gyula Fóra <[email protected]> 于2022年7月6日周三 13:56写道： > >> Here it is, copied from the docs essentially: >> >> FROM flink:1.15.0 >> >> >> # install python3: it has updated Python to 3.9 in Debian 11 and so >> install Python 3.7 from source >> # it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink >> officially. >> >> RUN apt-get update -y && \ >> apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev >> libffi-dev git && \ >> wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz && \ >> tar -xvf Python-3.7.9.tgz && \ >> cd Python-3.7.9 && \ >> ./configure --without-tests --enable-shared && \ >> make -j6 && \ >> make install && \ >> ldconfig /usr/local/lib && \ >> cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \ >> ln -s /usr/local/bin/python3 /usr/local/bin/python && \ >> apt-get clean && \ >> rm -rf /var/lib/apt/lists/* >> >> # install PyFlink >> RUN pip3 install apache-flink==1.15.0 >> >> And I am running: >> docker build --tag pyflink:latest . >> >> This gives the following error: >> >> >> *#6 64.12 cwd: /tmp/pip-install-9_farey_/pemja/#6 64.12 >> Complete output (1 lines):#6 64.12 Include folder should be at >> '/usr/local/openjdk-11/include' but doesn't exist. Please check you've >> installed the JDK properly.* >> >> A side note: >> The Dockerfile in the docs is missing git so initially I got the >> following error: >> >> *#7 57.73 raise OSError("%r was not found" % name)#7 57.73 >> OSError: 'git' was not found * >> >> @Weihua Hu <[email protected]> can you please send your working >> Dockerfile? >> >> Gyula >> >> On Wed, Jul 6, 2022 at 4:16 AM Weihua Hu <[email protected]> wrote: >> >>> Hi Gyula, >>> >>> I can build pyFlink image successfully by following this guide. Did you >>> add a dependency outside of the documentation? And could you provide your >>> Dockerfile >>> >>> >>> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker >>> >>> Best, >>> Weihua >>> >>> >>> On Tue, Jul 5, 2022 at 11:40 PM Gyula Fóra <[email protected]> wrote: >>> >>>> Well in any case either the official image is incorrect (maybe we >>>> should include JDK by default not JRE) or we should update the >>>> documentation regarding the python docker build because it simply doesn't >>>> work at the moment. >>>> >>>> I am still looking for a full working example that adds the required >>>> Python packages on top of a Flink 1.15.0 base image :) >>>> >>>> Gyula >>>> >>>> On Tue, Jul 5, 2022 at 5:36 PM Weihua Hu <[email protected]> >>>> wrote: >>>> >>>>> In addition, you can try providing the Dockerfile >>>>> >>>>> Best, >>>>> Weihua >>>>> >>>>> >>>>> On Tue, Jul 5, 2022 at 11:24 PM Weihua Hu <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> The base image flink:1.15.0 is built from openjdk:11-jre, and this >>>>>> image only installs jre but not jdk. >>>>>> It looks like the package you want to install (pemja) depends on jdk. >>>>>> you need install openjdk-11-jdk in dockerfile, >>>>>> take a look to how it is installed in the official image: >>>>>> >>>>>> >>>>>> https://hub.docker.com/layers/openjdk/library/openjdk/11-jdk/images/sha256-bc0af19c7c4f492fe6ff0c1d1c8c0e5dd90ab801385b220347bb91dbe2b4094f?context=explore >>>>>> >>>>>> >>>>>> Best, >>>>>> Weihua >>>>>> >>>>>> >>>>>> On Tue, Jul 5, 2022 at 3:50 PM Gyula Fóra <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All! >>>>>>> >>>>>>> I have been trying to experiment with the Flink python support on >>>>>>> Kubernetes but I got stuck creating a custom image with all the >>>>>>> necessary >>>>>>> python libraries. >>>>>>> >>>>>>> I found this guide in the docs: >>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker >>>>>>> >>>>>>> However when I try to build a custom image using it, I get the >>>>>>> following error: >>>>>>> >>>>>>> #7 131.7 Collecting pemja==0.1.4 >>>>>>> #7 131.8 Downloading pemja-0.1.4.tar.gz (32 kB) >>>>>>> #7 131.9 ERROR: Command errored out with exit status 255: >>>>>>> #7 131.9 command: /usr/local/bin/python3.7 -c 'import sys, >>>>>>> setuptools, tokenize; sys.argv[0] = >>>>>>> '"'"'/tmp/pip-install-y6o6djs1/pemja/setup.py'"'"'; >>>>>>> __file__='"'"'/tmp/pip-install-y6o6djs1/pemja/setup.py'"'"';f=getattr(tokenize, >>>>>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>>>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>>>>>> egg_info --egg-base /tmp/pip-pip-egg-info-47iid6nm >>>>>>> #7 131.9 cwd: /tmp/pip-install-y6o6djs1/pemja/ >>>>>>> #7 131.9 Complete output (1 lines): >>>>>>> #7 131.9 Include folder should be at >>>>>>> '/usr/local/openjdk-11/include' but doesn't exist. Please check you've >>>>>>> installed the JDK properly. >>>>>>> >>>>>>> Has anyone seen this before and knows the solution? >>>>>>> >>>>>>> Thanks >>>>>>> Gyula >>>>>>> >>>>>>
