Hi Gabor, Thanks a lot for the detailed explanation. Building a test environment with multi-platforms is indeed what we have been lacking, not only for M1 users, but also for Windows users. At 1.16, I remember that Azure did not provide the m1 environment, and the github action just planned m1 in the annual plan at that time[1][2].
+1 for drop 3.7 support in 1.19 [1] https://github.com/actions/runner-images/issues/2187 [2] https://github.com/github/roadmap/issues/528 Best, Xingbo Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 16:07写道： > Hi Xingbo, > > > So I guess you want to run ci tests on the m1 > > environment, but the current version of miniconda cannot meet this > > requirement, so there is a pre-step that must drop python 3.7? > > In short yes + local wheel build fix on the M1. In a bit more detailed > please let me elaborate. > > You're right about the arm64 build creation, Flink is doing that for quite > some time already and wheels are running fine. > There are gaps however. > > From user perspective: > * we're building wheels for ARM64 but we're not executing any python tests > on ARM64 but only on X64. > I think it's a must to test platform packages on the related platform since > not all python packages are ARM compatible or must be built/used in a > different way. > > From dev perspective: > * Local MacOs M1 wheel or python source distro builds are simply failing > with error messages. > Just to give an example GRPC pip install is failing and require the > following changes: > > https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32 > I can't really imagine how Azure is able to build MacOS ARM64 wheels... > * Python tests are not running on any kind of ARM CPUs which makes any bug > hunting/development hard or nearly impossible > > In order to give ARM support with local development + test execution > possibility new miniconda is needed. New miniconda runs 3.8+. > > There is already a PR which is under development: > https://github.com/apache/flink/pull/23359 > This fills all the mentioned gaps: local test/development/test execution on > ARM CPUs + the previously mentioned 3+ years not actively supported python > eliminated. > > Hope this helps. > > G > > > On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang <hxbks...@gmail.com> wrote: > > > Hi Gyala and Gabor, > > > > Pyflink has provided arm64 wheel packages for Apple silicon since > 1.16[1]. > > The use of Miniconda is only related to ci testing and packaging on linux > > platform, and building mac platform wheels are dependent on > > cibuildwheel[2]. So I guess you want to run ci tests on the m1 > > environment, but the current version of miniconda cannot meet this > > requirement, so there is a pre-step that must drop python 3.7? > > > > [1] https://pypi.org/project/apache-flink/1.16.2/#files > > [2] > > > > > https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30 > > > > Best, > > Xingbo > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 14:36写道： > > > > > Hi Xingbo, > > > > > > *Constraint:* > > > I personally not found any miniconda version which provides arm64 > support > > > together with python 3.7. > > > [image: image.png] > > > > > > At the moment I think new platform support means 3.7 drop. > > > > > > I fully to agree with Gyula, if we start now maybe we can release it in > > > half a year however *3.7 active support already ended in 27 Jun 2020*. > > > At the moment any python development/test execution on MacOS M1 is just > > > not working as-is just like any kind of python test execution on any > ARM > > > CPU. > > > > > > Gains: > > > * We can release a working version in half a year hopefully and not > > > shifting support to 1+ year > > > * MacOS M1 local development would work finally which is essential for > > > user engagement > > > * It would be possible to execute python tests on ARM64 machines > > > * We can shake up the python development story because it's not the > most > > > loved area > > > > > > BR, > > > G > > > > > > > > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra <gyula.f...@gmail.com> > wrote: > > > > > >> Hi Xingbo! > > >> > > >> I think we have to analyze what we gain by dropping 3.7 and upgrading > > to a > > >> miniconda version with a multiarch support. > > >> > > >> If this is what we need to get Apple silicon support then I think it's > > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even > > >> released > > >> yet so if we delay this to 1.20 that is basically 1 year from now. > > >> Making this change can increase the adoption instantly if we enable > new > > >> platforms. > > >> > > >> Cheers, > > >> Gyula > > >> > > >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang <hxbks...@gmail.com> > wrote: > > >> > > >> > Hi Gabor, > > >> > > > >> > Thanks for bringing this up. In my opinion, it is a bit aggressive > to > > >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1], > > >> and as > > >> > far as I know, many Pyflink users are still using python 3.7 as > their > > >> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 just > > like > > >> we > > >> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in 1.17[3]. > > >> > > > >> > For the support of Python 3.11, I am very supportive of the > > >> implementation > > >> > in 1.19 (many users have this appeal, and I originally wanted to > > >> support it > > >> > in 1.18). > > >> > > > >> > Regarding the miniconda upgrade, I tend to upgrade miniconda to the > > >> latest > > >> > version that can support python 3.7 to 3.11 at the same time. > > >> > > > >> > [1] https://w3techs.com/technologies/history_details/pl-python/3 > > >> > [2] https://issues.apache.org/jira/browse/FLINK-28195 > > >> > [3] https://issues.apache.org/jira/browse/FLINK-27929 > > >> > > > >> > Best, > > >> > Xingbo > > >> > > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年9月5日周二 04:10写道： > > >> > > > >> > > +1 > > >> > > > > >> > > @Dian should we add support of python 3.11 > > >> > > > > >> > > Best regards, > > >> > > Jing > > >> > > > > >> > > On Mon, Sep 4, 2023 at 3:39 PM Gabor Somogyi < > > >> gabor.g.somo...@gmail.com> > > >> > > wrote: > > >> > > > > >> > > > Thanks for all the responses! > > >> > > > > > >> > > > Based on the suggestions I've created the following jiras and > > >> started > > >> > to > > >> > > > work on them: > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33029 > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33030 > > >> > > > > > >> > > > The reason why I've split them is to separate the concerns and > > >> reduce > > >> > the > > >> > > > amount of code in a PR to help reviewers. > > >> > > > > > >> > > > BR, > > >> > > > G > > >> > > > > > >> > > > > > >> > > > On Mon, Sep 4, 2023 at 12:57 PM Sergey Nuyanzin < > > >> snuyan...@gmail.com> > > >> > > > wrote: > > >> > > > > > >> > > > > +1, > > >> > > > > Thanks for looking into this. > > >> > > > > > > >> > > > > On Mon, Sep 4, 2023 at 8:38 AM Gyula Fóra < > gyula.f...@gmail.com > > > > > >> > > wrote: > > >> > > > > > > >> > > > > > +1 > > >> > > > > > Thanks for looking into this. > > >> > > > > > > > >> > > > > > Gyula > > >> > > > > > > > >> > > > > > On Mon, Sep 4, 2023 at 8:26 AM Matthias Pohl < > > >> > matthias.p...@aiven.io > > >> > > > > > .invalid> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Thanks Gabor for looking into it. It sounds reasonable to > me > > >> as > > >> > > well. > > >> > > > > > > > > >> > > > > > > +1 > > >> > > > > > > > > >> > > > > > > On Sun, Sep 3, 2023 at 5:44 PM Márton Balassi < > > >> > > > > balassi.mar...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Hi Gabor, > > >> > > > > > > > > > >> > > > > > > > Thanks for bringing this up. Similarly to when we > dropped > > >> > Python > > >> > > > 3.6 > > >> > > > > > due > > >> > > > > > > to > > >> > > > > > > > its end of life (and added 3.10) in Flink 1.17 [1,2], it > > >> makes > > >> > > > sense > > >> > > > > to > > >> > > > > > > > proceed to remove 3.7 and add 3.11 instead. > > >> > > > > > > > > > >> > > > > > > > +1. > > >> > > > > > > > > > >> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-27929 > > >> > > > > > > > [2] https://github.com/apache/flink/pull/21699 > > >> > > > > > > > > > >> > > > > > > > Best, > > >> > > > > > > > Marton > > >> > > > > > > > > > >> > > > > > > > On Fri, Sep 1, 2023 at 10:39 AM Gabor Somogyi < > > >> > > > > > gabor.g.somo...@gmail.com > > >> > > > > > > > > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Hi All, > > >> > > > > > > > > > > >> > > > > > > > > I've analyzed through part of the pyflink code and > found > > >> some > > >> > > > > > > improvement > > >> > > > > > > > > possibilities. > > >> > > > > > > > > I would like to hear voices on the idea. > > >> > > > > > > > > > > >> > > > > > > > > Intention: > > >> > > > > > > > > * upgrade several python related versions to eliminate > > >> > > > end-of-life > > >> > > > > > > issues > > >> > > > > > > > > and keep up with bugfixes > > >> > > > > > > > > * start to add python arm64 support > > >> > > > > > > > > > > >> > > > > > > > > Actual situation: > > >> > > > > > > > > * Flink supports the following python versions: 3.7, > > 3.8, > > >> > 3.9, > > >> > > > 3.10 > > >> > > > > > > > > * We use miniconda 4.7.10 (python package management > > >> system > > >> > and > > >> > > > > > > > environment > > >> > > > > > > > > management system) which supports the following python > > >> > > versions: > > >> > > > > 3.7, > > >> > > > > > > > 3.8, > > >> > > > > > > > > 3.9, 3.10 > > >> > > > > > > > > * Our python framework is not supporting anything but > > >> x86_64 > > >> > > > > > > > > > > >> > > > > > > > > Issues: > > >> > > > > > > > > * Python 3.7.17 is the latest security patch of the > 3.7 > > >> line. > > >> > > > This > > >> > > > > > > > version > > >> > > > > > > > > is end-of-life and is no longer supported: > > >> > > > > > > > > https://www.python.org/downloads/release/python-3717/ > > >> > > > > > > > > * Miniconda 4.7.10 is released on 2019-07-29 which is > 4 > > >> years > > >> > > old > > >> > > > > > > already > > >> > > > > > > > > and not supporting too many architectures (x86_64 and > > >> > ppc64le) > > >> > > > > > > > > * The latest miniconda which has real multi-arch > feature > > >> set > > >> > > > > supports > > >> > > > > > > the > > >> > > > > > > > > following python versions: 3.8, 3.9, 3.10, 3.11 and no > > 3.7 > > >> > > > support > > >> > > > > > > > > > > >> > > > > > > > > Suggestion to solve the issues: > > >> > > > > > > > > * In 1.19 drop python 3.7 support and upgrade > miniconda > > to > > >> > the > > >> > > > > latest > > >> > > > > > > > > version which opens the door to other platform + > python > > >> 3.11 > > >> > > > > support > > >> > > > > > > > > > > >> > > > > > > > > Please note python 3.11 support is not > > initiated/discussed > > >> > > here. > > >> > > > > > > > > > > >> > > > > > > > > BR, > > >> > > > > > > > > G > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > -- > > >> > > > > Best regards, > > >> > > > > Sergey > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
