Yes Ahmet. That would be great. There are some load tests defined in the https://github.com/apache/beam/blob/master/.test-infra which could be useful for performance testing of Beam between 3.10 and 3.11. Do you suggest any other tests?
On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote: > Thank you, this is great! > > Python 3.11 announcement had a claim about performance [1]: > > "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured > with the pyperformance benchmark suite, when compiled with GCC on Ubuntu > Linux. Depending on your workload, the overall speedup could be 10-60%." > > Have we measured this in Beam? Are we seeing any benefits? If not, why? If > yes, this would be a cool blog post as well. > > Ahmet > > > On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <dev@beam.apache.org> > wrote: > >> Python 3.11 support has been merged at >> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release. >> >> Please let me know if you have any questions. >> >> Thanks, >> Anand >> >> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Thanks a lot Anand. I'll take a look at the PRs. >>> >>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <ananding...@google.com> >>> wrote: >>> >>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599 >>>> that updates the build dependencies of Apache Beam. >>>> >>>> Several GCP dependencies needed to be updated as well. I covered them >>>> in the PR: https://github.com/apache/beam/pull/24599 >>>> >>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <ananding...@google.com> >>>> wrote: >>>> >>>>> Yes, we may need to update all of them >>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>. >>>>> I can add more information once I dig into the issue(most likely next >>>>> week). I will comment on my findings on the issue: >>>>> https://github.com/apache/beam/issues/24569 and will periodically >>>>> update this thread. >>>>> >>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <ananding...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Yes, it is related to protobuf only. But I think the update of these >>>>>>> dependencies are required for Python 3.11 since the newer versions have >>>>>>> support for Python 3.11 wheels. >>>>>>> >>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for >>>>>> protobuf==3.x.x and that can cause friction. >>>>>> https://pypi.org/project/protobuf/3.20.3/#files >>>>>> >>>>>> I would probably narrow the problem further to demonstrate which >>>>>> stubs are not being generated, and if reason not obvious we can also ask >>>>>> for feedback from protobuf maintainers. Also - do we by chance need to >>>>>> update some other deps from >>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33 >>>>>> for this to work? >>>>>> >>>>>> Also: tracking issue for protobuf4 support in Beam: >>>>>> https://github.com/apache/beam/issues/24569. >>>>>> >>>>>> If we use older versions of these packages, then we have to depend on >>>>>>> installing those packages on Python 3.11 from source distributions >>>>>>> which is >>>>>>> not desired. >>>>>>> >>>>>>> I am working parallely on that issue in a different PR >>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue >>>>>>> should be a blocker for Python 3.11 update. >>>>>>> >>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>>> Hi Anand, >>>>>>>> >>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> We are planning to work on adding support for Python 3.11[1] to >>>>>>>>> Apache Beam Python SDK. >>>>>>>>> >>>>>>>>> As part of this effort, we are going to update the python build >>>>>>>>> dependencies defined at [2]. >>>>>>>>> >>>>>>>>> Right now, there is an error with the newer version of >>>>>>>>> protobuf(4.21.11). It is not generating _urn files. >>>>>>>>> >>>>>>>>> It can be reproduced by >>>>>>>>> >>>>>>>> >>>>>>>>> 1. python setup.py sdist >>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz >>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam >>>>>>>>> >>>>>>>> I think the error you are describing is related to protobuf 4, so >>>>>>>> the repro should focus on the portion where generation of stubs is >>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + >>>>>>>> Python >>>>>>>> 3.11? >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> will lead to *ImportError: cannot import name >>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. >>>>>>>>> *Running >>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help >>>>>>>>> either. >>>>>>>>> >>>>>>>>> If you have encountered this error and found a resolution, please >>>>>>>>> let me know(that would be super helpful). >>>>>>>>> >>>>>>>>> I am going to work on this soon. Please let me know if you want to >>>>>>>>> collaborate. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Anand Inguva >>>>>>>>> >>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721 >>>>>>>>> [2] >>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt >>>>>>>>> >>>>>>>>
