+1 (binding) Tested with https://github.com/Talend/beam-samples/ (Java SDK v8/v11/v17, Spark 3 runner).
--- Alexey > On 11 Jan 2023, at 16:53, Ritesh Ghorse via dev <dev@beam.apache.org> wrote: > > +1 (non-binding) > Validated Go Dataframe Transform wrapper on Dataflow runner and Go SDK > quickstart on Direct and Dataflow Runner. > > Thanks! > > On Wed, Jan 11, 2023 at 12:51 AM Anand Inguva via dev <dev@beam.apache.org > <mailto:dev@beam.apache.org>> wrote: >> I ran the Python word count on DirectRunner and Dataflow Runner. >> >> Steps: >> 1. pip install --pre apache_beam in a fresh virtualenv. >> 2. Run the command Ahmet provided except removing the sdk_location from CMD >> args. >> >> The job was successful. <image.png> >> >> On Tue, Jan 10, 2023 at 6:48 PM Ahmet Altay via dev <dev@beam.apache.org >> <mailto:dev@beam.apache.org>> wrote: >>> I validated python quick starts (direct, dataflow) X (batch, streaming). I >>> ran into an issue with the dataflow batch case, running the wordcount with >>> the standard: >>> >>> python -m apache_beam.examples.wordcount \ >>> --output <GCS> \ >>> --staging_location <GCS> \ >>> --temp_location <GCS>\ >>> --runner DataflowRunner \ >>> --job_name wordcount-$USER \ >>> --project <PROJECT> \ >>> --num_workers 1 \ >>> --region us-central1 \ >>> --sdk_location apache-beam-2.44.0.zip >>> >>> results in: >>> >>> "/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py", line >>> 589, in __enter__ raise RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE) >>> RuntimeError: This pipeline requires Dataflow Runner v2 in order to run >>> with currently used version of Apache Beam on Python 3.10+. Please verify >>> that the Dataflow Runner v2 is not disabled in the pipeline options or >>> enable it explicitly via: --dataflow_service_option=use_runner_v2. >>> Alternatively, downgrade to Python 3.9 to use Dataflow Runner v1. >>> >>> Questions: >>> - I am not explicitly opting out of runner v2, and this is a standard >>> wordcount example, I expected it to just work. >>> >>> Then I tried to add --dataflow_service_option=use_runner_v2 to the above >>> wordcount command, which results in the following error: >>> >>> "message": "Dataflow Runner v2 requires a valid FnApi job, Please >>> resubmit your job with a valid configuration. Note that if using Templates, >>> you may need to regenerate your template with the '--use_runner_v2'." >>> >>> Maybe I am doing something wrong and it is an error on my end. It would be >>> good for someone else with python experience to check this. >>> >>> /cc @Valentyn Tymofieiev <mailto:valen...@google.com> >>> >>> Ahmet >>> >>> >>> >>> >>> On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles <k...@apache.org >>> <mailto:k...@apache.org>> wrote: >>>> I have published a new maven staging repository: >>>> https://repository.apache.org/content/repositories/orgapachebeam-1290/ >>>> >>>> It looks like it has everything, though I did not automate a check. At >>>> least there were no errors during publish which I ran with --no-parallel >>>> overnight, and some specific things that were missing from >>>> orgapachebeam-1289 are present. >>>> >>>> I will restart the 72 hour waiting period, since the RC is only now usable. >>>> >>>> Kenn >>>> >>>> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles <k...@apache.org >>>> <mailto:k...@apache.org>> wrote: >>>>> I have discovered that many pom files are missing from the nexus >>>>> repository. I should be able to re-publish a new one. It will take some >>>>> time as this is one of the longest-running processes. >>>>> >>>>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles <k...@apache.org >>>>> <mailto:k...@apache.org>> wrote: >>>>>> Correction: this is release candidate #1. >>>>>> >>>>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles <k...@apache.org >>>>>> <mailto:k...@apache.org>> wrote: >>>>>>> Hi everyone, >>>>>>> >>>>>>> Please review and vote on the release candidate #3 for the version >>>>>>> 2.44.0, as follows: >>>>>>> [ ] +1, Approve the release >>>>>>> [ ] -1, Do not approve the release (please provide specific comments) >>>>>>> >>>>>>> Reviewers are encouraged to test their own use cases with the release >>>>>>> candidate, and vote +1 if >>>>>>> no issues are found. >>>>>>> >>>>>>> The complete staging area is available for your review, which includes: >>>>>>> * GitHub Release notes [1], >>>>>>> * the official Apache source release to be deployed to dist.apache.org >>>>>>> <http://dist.apache.org/> [2], which is signed with the key with >>>>>>> fingerprint 6ED551A8AE02461C [3], >>>>>>> * all artifacts to be deployed to the Maven Central Repository [4], >>>>>>> * source code tag "v2.44.0-RC1" [5], >>>>>>> * website pull request listing the release [6], the blog post [6], and >>>>>>> publishing the API reference manual [7]. >>>>>>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_232. >>>>>>> * Python artifacts are deployed along with the source release to the >>>>>>> dist.apache.org <http://dist.apache.org/> [2] and PyPI [8]. >>>>>>> * Go artifacts and documentation are available at pkg.go.dev >>>>>>> <http://pkg.go.dev/> [9] (waiting on these to appear) >>>>>>> * Validation sheet with a tab for 2.44.0 release to help with >>>>>>> validation [10]. >>>>>>> * Docker images published to Docker Hub [11]. >>>>>>> >>>>>>> The vote will be open for at least 72 hours. It is adopted by majority >>>>>>> approval, with at least 3 PMC affirmative votes. >>>>>>> >>>>>>> For guidelines on how to try the release in your projects, check out >>>>>>> our blog post at /blog/validate-beam-release/. >>>>>>> >>>>>>> Thanks, >>>>>>> Kenn >>>>>>> >>>>>>> [1] https://github.com/apache/beam/milestone/7 >>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.44.0/ >>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS >>>>>>> [4] >>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1289/ >>>>>>> [5] https://github.com/apache/beam/tree/v2.44.0-RC1 >>>>>>> [6] https://github.com/apache/beam/pull/24951 >>>>>>> [7] https://github.com/apache/beam-site/pull/638 >>>>>>> [8] https://pypi.org/project/apache-beam/2.44.0rc1/ >>>>>>> [9] >>>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam >>>>>>> [10] >>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910 >>>>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image
