Thank you all for the questions/suggestions! After some questions/clarifications there are only +1 on the discussion. Unless there are further questions/objections I'm intended to merge the related PR in 24 hours: https://github.com/apache/flink/pull/23359
G On Wed, Sep 6, 2023 at 10:53 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Hi Xingbo, > > Thanks for your support! > > I agree that multi-platform in general is a good idea. We start with ARM > but this opens the door for all others. > > G > > > On Wed, Sep 6, 2023 at 10:46 AM Xingbo Huang <hxbks...@gmail.com> wrote: > >> Hi Gabor, >> >> Thanks a lot for the detailed explanation. Building a test environment >> with >> multi-platforms is indeed what we have been lacking, >> not only for M1 users, but also for Windows users. At 1.16, I remember >> that >> Azure did not provide the m1 environment, >> and the github action just planned m1 in the annual plan at that >> time[1][2]. >> >> +1 for drop 3.7 support in 1.19 >> >> [1] https://github.com/actions/runner-images/issues/2187 >> [2] https://github.com/github/roadmap/issues/528 >> >> Best, >> Xingbo >> >> Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 16:07写道： >> >> > Hi Xingbo, >> > >> > > So I guess you want to run ci tests on the m1 >> > > environment, but the current version of miniconda cannot meet this >> > > requirement, so there is a pre-step that must drop python 3.7? >> > >> > In short yes + local wheel build fix on the M1. In a bit more detailed >> > please let me elaborate. >> > >> > You're right about the arm64 build creation, Flink is doing that for >> quite >> > some time already and wheels are running fine. >> > There are gaps however. >> > >> > From user perspective: >> > * we're building wheels for ARM64 but we're not executing any python >> tests >> > on ARM64 but only on X64. >> > I think it's a must to test platform packages on the related platform >> since >> > not all python packages are ARM compatible or must be built/used in a >> > different way. >> > >> > From dev perspective: >> > * Local MacOs M1 wheel or python source distro builds are simply failing >> > with error messages. >> > Just to give an example GRPC pip install is failing and require the >> > following changes: >> > >> > >> https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32 >> > I can't really imagine how Azure is able to build MacOS ARM64 wheels... >> > * Python tests are not running on any kind of ARM CPUs which makes any >> bug >> > hunting/development hard or nearly impossible >> > >> > In order to give ARM support with local development + test execution >> > possibility new miniconda is needed. New miniconda runs 3.8+. >> > >> > There is already a PR which is under development: >> > https://github.com/apache/flink/pull/23359 >> > This fills all the mentioned gaps: local test/development/test >> execution on >> > ARM CPUs + the previously mentioned 3+ years not actively supported >> python >> > eliminated. >> > >> > Hope this helps. >> > >> > G >> > >> > >> > On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang <hxbks...@gmail.com> wrote: >> > >> > > Hi Gyala and Gabor, >> > > >> > > Pyflink has provided arm64 wheel packages for Apple silicon since >> > 1.16[1]. >> > > The use of Miniconda is only related to ci testing and packaging on >> linux >> > > platform, and building mac platform wheels are dependent on >> > > cibuildwheel[2]. So I guess you want to run ci tests on the m1 >> > > environment, but the current version of miniconda cannot meet this >> > > requirement, so there is a pre-step that must drop python 3.7? >> > > >> > > [1] https://pypi.org/project/apache-flink/1.16.2/#files >> > > [2] >> > > >> > > >> > >> https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30 >> > > >> > > Best, >> > > Xingbo >> > > >> > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 14:36写道： >> > > >> > > > Hi Xingbo, >> > > > >> > > > *Constraint:* >> > > > I personally not found any miniconda version which provides arm64 >> > support >> > > > together with python 3.7. >> > > > [image: image.png] >> > > > >> > > > At the moment I think new platform support means 3.7 drop. >> > > > >> > > > I fully to agree with Gyula, if we start now maybe we can release >> it in >> > > > half a year however *3.7 active support already ended in 27 Jun >> 2020*. >> > > > At the moment any python development/test execution on MacOS M1 is >> just >> > > > not working as-is just like any kind of python test execution on any >> > ARM >> > > > CPU. >> > > > >> > > > Gains: >> > > > * We can release a working version in half a year hopefully and not >> > > > shifting support to 1+ year >> > > > * MacOS M1 local development would work finally which is essential >> for >> > > > user engagement >> > > > * It would be possible to execute python tests on ARM64 machines >> > > > * We can shake up the python development story because it's not the >> > most >> > > > loved area >> > > > >> > > > BR, >> > > > G >> > > > >> > > > >> > > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra <gyula.f...@gmail.com> >> > wrote: >> > > > >> > > >> Hi Xingbo! >> > > >> >> > > >> I think we have to analyze what we gain by dropping 3.7 and >> upgrading >> > > to a >> > > >> miniconda version with a multiarch support. >> > > >> >> > > >> If this is what we need to get Apple silicon support then I think >> it's >> > > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even >> > > >> released >> > > >> yet so if we delay this to 1.20 that is basically 1 year from now. >> > > >> Making this change can increase the adoption instantly if we enable >> > new >> > > >> platforms. >> > > >> >> > > >> Cheers, >> > > >> Gyula >> > > >> >> > > >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang <hxbks...@gmail.com> >> > wrote: >> > > >> >> > > >> > Hi Gabor, >> > > >> > >> > > >> > Thanks for bringing this up. In my opinion, it is a bit >> aggressive >> > to >> > > >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a >> lot[1], >> > > >> and as >> > > >> > far as I know, many Pyflink users are still using python 3.7 as >> > their >> > > >> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 >> just >> > > like >> > > >> we >> > > >> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in >> 1.17[3]. >> > > >> > >> > > >> > For the support of Python 3.11, I am very supportive of the >> > > >> implementation >> > > >> > in 1.19 (many users have this appeal, and I originally wanted to >> > > >> support it >> > > >> > in 1.18). >> > > >> > >> > > >> > Regarding the miniconda upgrade, I tend to upgrade miniconda to >> the >> > > >> latest >> > > >> > version that can support python 3.7 to 3.11 at the same time. >> > > >> > >> > > >> > [1] https://w3techs.com/technologies/history_details/pl-python/3 >> > > >> > [2] https://issues.apache.org/jira/browse/FLINK-28195 >> > > >> > [3] https://issues.apache.org/jira/browse/FLINK-27929 >> > > >> > >> > > >> > Best, >> > > >> > Xingbo >> > > >> > >> > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年9月5日周二 04:10写道： >> > > >> > >> > > >> > > +1 >> > > >> > > >> > > >> > > @Dian should we add support of python 3.11 >> > > >> > > >> > > >> > > Best regards, >> > > >> > > Jing >> > > >> > > >> > > >> > > On Mon, Sep 4, 2023 at 3:39 PM Gabor Somogyi < >> > > >> gabor.g.somo...@gmail.com> >> > > >> > > wrote: >> > > >> > > >> > > >> > > > Thanks for all the responses! >> > > >> > > > >> > > >> > > > Based on the suggestions I've created the following jiras and >> > > >> started >> > > >> > to >> > > >> > > > work on them: >> > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33029 >> > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33030 >> > > >> > > > >> > > >> > > > The reason why I've split them is to separate the concerns >> and >> > > >> reduce >> > > >> > the >> > > >> > > > amount of code in a PR to help reviewers. >> > > >> > > > >> > > >> > > > BR, >> > > >> > > > G >> > > >> > > > >> > > >> > > > >> > > >> > > > On Mon, Sep 4, 2023 at 12:57 PM Sergey Nuyanzin < >> > > >> snuyan...@gmail.com> >> > > >> > > > wrote: >> > > >> > > > >> > > >> > > > > +1, >> > > >> > > > > Thanks for looking into this. >> > > >> > > > > >> > > >> > > > > On Mon, Sep 4, 2023 at 8:38 AM Gyula Fóra < >> > gyula.f...@gmail.com >> > > > >> > > >> > > wrote: >> > > >> > > > > >> > > >> > > > > > +1 >> > > >> > > > > > Thanks for looking into this. >> > > >> > > > > > >> > > >> > > > > > Gyula >> > > >> > > > > > >> > > >> > > > > > On Mon, Sep 4, 2023 at 8:26 AM Matthias Pohl < >> > > >> > matthias.p...@aiven.io >> > > >> > > > > > .invalid> >> > > >> > > > > > wrote: >> > > >> > > > > > >> > > >> > > > > > > Thanks Gabor for looking into it. It sounds reasonable >> to >> > me >> > > >> as >> > > >> > > well. >> > > >> > > > > > > >> > > >> > > > > > > +1 >> > > >> > > > > > > >> > > >> > > > > > > On Sun, Sep 3, 2023 at 5:44 PM Márton Balassi < >> > > >> > > > > balassi.mar...@gmail.com> >> > > >> > > > > > > wrote: >> > > >> > > > > > > >> > > >> > > > > > > > Hi Gabor, >> > > >> > > > > > > > >> > > >> > > > > > > > Thanks for bringing this up. Similarly to when we >> > dropped >> > > >> > Python >> > > >> > > > 3.6 >> > > >> > > > > > due >> > > >> > > > > > > to >> > > >> > > > > > > > its end of life (and added 3.10) in Flink 1.17 >> [1,2], it >> > > >> makes >> > > >> > > > sense >> > > >> > > > > to >> > > >> > > > > > > > proceed to remove 3.7 and add 3.11 instead. >> > > >> > > > > > > > >> > > >> > > > > > > > +1. >> > > >> > > > > > > > >> > > >> > > > > > > > [1] >> https://issues.apache.org/jira/browse/FLINK-27929 >> > > >> > > > > > > > [2] https://github.com/apache/flink/pull/21699 >> > > >> > > > > > > > >> > > >> > > > > > > > Best, >> > > >> > > > > > > > Marton >> > > >> > > > > > > > >> > > >> > > > > > > > On Fri, Sep 1, 2023 at 10:39 AM Gabor Somogyi < >> > > >> > > > > > gabor.g.somo...@gmail.com >> > > >> > > > > > > > >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > >> > > >> > > > > > > > > Hi All, >> > > >> > > > > > > > > >> > > >> > > > > > > > > I've analyzed through part of the pyflink code and >> > found >> > > >> some >> > > >> > > > > > > improvement >> > > >> > > > > > > > > possibilities. >> > > >> > > > > > > > > I would like to hear voices on the idea. >> > > >> > > > > > > > > >> > > >> > > > > > > > > Intention: >> > > >> > > > > > > > > * upgrade several python related versions to >> eliminate >> > > >> > > > end-of-life >> > > >> > > > > > > issues >> > > >> > > > > > > > > and keep up with bugfixes >> > > >> > > > > > > > > * start to add python arm64 support >> > > >> > > > > > > > > >> > > >> > > > > > > > > Actual situation: >> > > >> > > > > > > > > * Flink supports the following python versions: >> 3.7, >> > > 3.8, >> > > >> > 3.9, >> > > >> > > > 3.10 >> > > >> > > > > > > > > * We use miniconda 4.7.10 (python package >> management >> > > >> system >> > > >> > and >> > > >> > > > > > > > environment >> > > >> > > > > > > > > management system) which supports the following >> python >> > > >> > > versions: >> > > >> > > > > 3.7, >> > > >> > > > > > > > 3.8, >> > > >> > > > > > > > > 3.9, 3.10 >> > > >> > > > > > > > > * Our python framework is not supporting anything >> but >> > > >> x86_64 >> > > >> > > > > > > > > >> > > >> > > > > > > > > Issues: >> > > >> > > > > > > > > * Python 3.7.17 is the latest security patch of the >> > 3.7 >> > > >> line. >> > > >> > > > This >> > > >> > > > > > > > version >> > > >> > > > > > > > > is end-of-life and is no longer supported: >> > > >> > > > > > > > > >> https://www.python.org/downloads/release/python-3717/ >> > > >> > > > > > > > > * Miniconda 4.7.10 is released on 2019-07-29 which >> is >> > 4 >> > > >> years >> > > >> > > old >> > > >> > > > > > > already >> > > >> > > > > > > > > and not supporting too many architectures (x86_64 >> and >> > > >> > ppc64le) >> > > >> > > > > > > > > * The latest miniconda which has real multi-arch >> > feature >> > > >> set >> > > >> > > > > supports >> > > >> > > > > > > the >> > > >> > > > > > > > > following python versions: 3.8, 3.9, 3.10, 3.11 >> and no >> > > 3.7 >> > > >> > > > support >> > > >> > > > > > > > > >> > > >> > > > > > > > > Suggestion to solve the issues: >> > > >> > > > > > > > > * In 1.19 drop python 3.7 support and upgrade >> > miniconda >> > > to >> > > >> > the >> > > >> > > > > latest >> > > >> > > > > > > > > version which opens the door to other platform + >> > python >> > > >> 3.11 >> > > >> > > > > support >> > > >> > > > > > > > > >> > > >> > > > > > > > > Please note python 3.11 support is not >> > > initiated/discussed >> > > >> > > here. >> > > >> > > > > > > > > >> > > >> > > > > > > > > BR, >> > > >> > > > > > > > > G >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > -- >> > > >> > > > > Best regards, >> > > >> > > > > Sergey >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > > >> > > >> > >> >
