Thank you, this is great! Python 3.11 announcement had a claim about performance [1]:
"CPython 3.11 is an average of 25% faster than CPython 3.10 as measured with the pyperformance benchmark suite, when compiled with GCC on Ubuntu Linux. Depending on your workload, the overall speedup could be 10-60%." Have we measured this in Beam? Are we seeing any benefits? If not, why? If yes, this would be a cool blog post as well. Ahmet On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <dev@beam.apache.org> wrote: > Python 3.11 support has been merged at > https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release. > > Please let me know if you have any questions. > > Thanks, > Anand > > On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Thanks a lot Anand. I'll take a look at the PRs. >> >> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <ananding...@google.com> >> wrote: >> >>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599 >>> that updates the build dependencies of Apache Beam. >>> >>> Several GCP dependencies needed to be updated as well. I covered them in >>> the PR: https://github.com/apache/beam/pull/24599 >>> >>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <ananding...@google.com> >>> wrote: >>> >>>> Yes, we may need to update all of them >>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>. >>>> I can add more information once I dig into the issue(most likely next >>>> week). I will comment on my findings on the issue: >>>> https://github.com/apache/beam/issues/24569 and will periodically >>>> update this thread. >>>> >>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <valen...@google.com> >>>> wrote: >>>> >>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <ananding...@google.com> >>>>> wrote: >>>>> >>>>>> Yes, it is related to protobuf only. But I think the update of these >>>>>> dependencies are required for Python 3.11 since the newer versions have >>>>>> support for Python 3.11 wheels. >>>>>> >>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for >>>>> protobuf==3.x.x and that can cause friction. >>>>> https://pypi.org/project/protobuf/3.20.3/#files >>>>> >>>>> I would probably narrow the problem further to demonstrate which stubs >>>>> are not being generated, and if reason not obvious we can also ask for >>>>> feedback from protobuf maintainers. Also - do we by chance need to >>>>> update some other deps from >>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33 >>>>> for this to work? >>>>> >>>>> Also: tracking issue for protobuf4 support in Beam: >>>>> https://github.com/apache/beam/issues/24569. >>>>> >>>>> If we use older versions of these packages, then we have to depend on >>>>>> installing those packages on Python 3.11 from source distributions which >>>>>> is >>>>>> not desired. >>>>>> >>>>>> I am working parallely on that issue in a different PR >>>>>> https://github.com/apache/beam/pull/24599 but I think this issue >>>>>> should be a blocker for Python 3.11 update. >>>>>> >>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev < >>>>>> valen...@google.com> wrote: >>>>>> >>>>>>> Hi Anand, >>>>>>> >>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev < >>>>>>> dev@beam.apache.org> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We are planning to work on adding support for Python 3.11[1] to >>>>>>>> Apache Beam Python SDK. >>>>>>>> >>>>>>>> As part of this effort, we are going to update the python build >>>>>>>> dependencies defined at [2]. >>>>>>>> >>>>>>>> Right now, there is an error with the newer version of >>>>>>>> protobuf(4.21.11). It is not generating _urn files. >>>>>>>> >>>>>>>> It can be reproduced by >>>>>>>> >>>>>>> >>>>>>>> 1. python setup.py sdist >>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz >>>>>>>> 3. switch to python interpreter and run import apache_beam as beam >>>>>>>> >>>>>>> I think the error you are describing is related to protobuf 4, so >>>>>>> the repro should focus on the portion where generation of stubs is >>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + >>>>>>> Python >>>>>>> 3.11? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> will lead to *ImportError: cannot import name >>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. >>>>>>>> *Running >>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either. >>>>>>>> >>>>>>>> If you have encountered this error and found a resolution, please >>>>>>>> let me know(that would be super helpful). >>>>>>>> >>>>>>>> I am going to work on this soon. Please let me know if you want to >>>>>>>> collaborate. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Anand Inguva >>>>>>>> >>>>>>>> *[1] *https://github.com/apache/beam/pull/24721 >>>>>>>> [2] >>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt >>>>>>>> >>>>>>>
