Those particular errors are often expected in the sink due to the protocol used. If a work item retries before committing (which could happen for many reasons including worker crashes), it will experience those errors.
On Fri, Apr 28, 2023 at 12:55 PM Ahmed Abualsaud <ahmedabuals...@google.com> wrote: > @Danny McCormick <dannymccorm...@google.com> @Reuven Lax > <re...@google.com> sorry it's been a while since you looked into this, > but do you remember if the fix in #25642 > <https://github.com/apache/beam/pull/25642> issue is related to the > recent "ALREADY_EXISTS: The offset is within stream, expected offset..." > errors? > > On Fri, Mar 10, 2023 at 7:47 PM Ahmet Altay via dev <dev@beam.apache.org> > wrote: > >> Thank you! >> >> Is there a tracking issue for this known issue? And would the known >> issues section of the release notes link to that? >> >> >> On Fri, Mar 10, 2023 at 11:38 AM Danny McCormick via dev < >> dev@beam.apache.org> wrote: >> >>> We determined that the same issue exists in the 2.45 release, so we are >>> going to continue finalizing the release candidate. Thank you for your >>> patience. >>> >>> Thanks, >>> Danny >>> >>> On Wed, Mar 8, 2023 at 3:15 PM Reuven Lax <re...@google.com> wrote: >>> >>>> We are trying to reproduce and debug the issue we saw to validate >>>> whether it was a real regression or not. Will update when we know more. >>>> >>>> On Wed, Mar 8, 2023 at 11:31 AM Danny McCormick < >>>> dannymccorm...@google.com> wrote: >>>> >>>>> >>>>> @Reuven Lax <re...@google.com> found a new potential regression in >>>>> BigQuery I/O, so I have paused the release rollout. I had already pushed >>>>> the Python artifacts and Go tags, but not the Java ones. We have since >>>>> temporarily yanked <https://pypi.org/help/#yanked> the Python release >>>>> and deleted the Go tags, they were live for around an hour. The possible >>>>> regression is in Java, so neither of those releases should be affected, >>>>> but >>>>> x-lang may not work properly because it depends on versioning. I will >>>>> update this thread with next steps when we know more. >>>>> >>>>> Thanks, >>>>> Danny >>>>> On Wed, Mar 8, 2023 at 5:59 AM Jan Lukavský <je...@seznam.cz> wrote: >>>>> >>>>>> +1 (binding) >>>>>> >>>>>> Tested Java SDK with Flink and Spark 3 runner. >>>>>> >>>>>> Thanks, >>>>>> Jan >>>>>> >>>>>> On 3/8/23 01:53, Valentyn Tymofieiev via dev wrote: >>>>>> >>>>>> +1. Verified the composition of Python containers and ran Python >>>>>> pipelines on Dataflow runner v1 and runner v2. >>>>>> >>>>>> On Tue, Mar 7, 2023 at 4:11 PM Ritesh Ghorse via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> +1 (non-binding) >>>>>>> Validated Go SDK quickstart on direct and dataflow runner >>>>>>> >>>>>>> On Tue, Mar 7, 2023 at 10:54 AM Alexey Romanenko < >>>>>>> aromanenko....@gmail.com> wrote: >>>>>>> >>>>>>>> +1 (binding) >>>>>>>> >>>>>>>> Tested with https://github.com/Talend/beam-samples/ >>>>>>>> (Java SDK v8/v11/v17, Spark 3.x runner). >>>>>>>> >>>>>>>> --- >>>>>>>> Alexey >>>>>>>> >>>>>>>> On 7 Mar 2023, at 07:38, Ahmet Altay via dev <dev@beam.apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>> +1 (binding) - I validated python quickstarts on direct & dataflow >>>>>>>> runners. >>>>>>>> >>>>>>>> Thank you for doing the release! >>>>>>>> >>>>>>>> On Sat, Mar 4, 2023 at 8:01 AM Chamikara Jayalath via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> +1 (binding) >>>>>>>>> >>>>>>>>> Validated multi-language Java and Python pipelines. >>>>>>>>> >>>>>>>>> On Fri, Mar 3, 2023 at 1:59 PM Danny McCormick via dev < >>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>> >>>>>>>>>> > I have encountered a failure in a Python pipeline running with >>>>>>>>>> Runner v1: >>>>>>>>>> >>>>>>>>>> > RuntimeError: Beam SDK base version 2.46.0 does not match >>>>>>>>>> Dataflow Python worker version 2.45.0. Please check Dataflow worker >>>>>>>>>> startup >>>>>>>>>> logs and make sure that correct version of Beam SDK is installed. >>>>>>>>>> >>>>>>>>>> > We should understand why Python ValidatesRunner tests (which >>>>>>>>>> have passed) didn't catch this error. >>>>>>>>>> >>>>>>>>>> > This can be remediated in Dataflow containers without changes >>>>>>>>>> to the release candidate. >>>>>>>>>> >>>>>>>>>> Good catch! I've kicked off a release to fix this, it should be >>>>>>>>>> done later this evening - I won't be available when it completes, >>>>>>>>>> but I >>>>>>>>>> would expect it to be around 5:00 PST. >>>>>>>>>> >>>>>>>>>> On Fri, Mar 3, 2023 at 3:49 PM Danny McCormick < >>>>>>>>>> dannymccorm...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Reuven, could you provide some more context on the bug/why >>>>>>>>>>> it is important? Does it meet the standard in >>>>>>>>>>> https://beam.apache.org/contribute/release-guide/#7-triage-release-blocking-issues-in-github? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The release branch was cut last Wednesday, so that is why it is >>>>>>>>>>> not included. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Seems like this was a revert of a previous commit that was also >>>>>>>>> not included in the 2.46.0 release branch ( >>>>>>>>> https://github.com/apache/beam/pull/25627) ? >>>>>>>>> >>>>>>>>> If so we might not need a new RC but good to confirm. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Cham >>>>>>>>> >>>>>>>>> >>>>>>>>>>> On Fri, Mar 3, 2023 at 3:24 PM Reuven Lax <re...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> If possible, I would like to see if we could include >>>>>>>>>>>> https://github.com/apache/beam/pull/25642 as we believe this >>>>>>>>>>>> bug has been impacting multiple users. This was merged 4 days ago, >>>>>>>>>>>> but this >>>>>>>>>>>> RC cut does not seem to include it. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Mar 3, 2023 at 12:18 PM Valentyn Tymofieiev via dev < >>>>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I have encountered a failure in a Python pipeline running with >>>>>>>>>>>>> Runner v1: >>>>>>>>>>>>> >>>>>>>>>>>>> RuntimeError: Beam SDK base version 2.46.0 does not match >>>>>>>>>>>>> Dataflow Python worker version 2.45.0. Please check Dataflow >>>>>>>>>>>>> worker startup >>>>>>>>>>>>> logs and make sure that correct version of Beam SDK is installed. >>>>>>>>>>>>> >>>>>>>>>>>>> We should understand why Python ValidatesRunner tests (which >>>>>>>>>>>>> have passed) didn't catch this error. >>>>>>>>>>>>> >>>>>>>>>>>>> This can be remediated in Dataflow containers without changes >>>>>>>>>>>>> to the release candidate. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Mar 3, 2023 at 11:22 AM Robert Bradshaw via dev < >>>>>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> +1 (binding). >>>>>>>>>>>>>> >>>>>>>>>>>>>> I verified that the artifacts and signatures all look good, >>>>>>>>>>>>>> all the >>>>>>>>>>>>>> containers are pushed, and tested some pipelines with a fresh >>>>>>>>>>>>>> install >>>>>>>>>>>>>> from one of the Python wheels. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Mar 3, 2023 at 11:13 AM Danny McCormick >>>>>>>>>>>>>> <dannymccorm...@google.com> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > > The released artifacts seem to be missing the last commit >>>>>>>>>>>>>> at >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> https://github.com/apache/beam/commit/c528eab18b32342daed53b750fe330d30c7e5224 >>>>>>>>>>>>>> > > . Is this essential to the release, or just useful for >>>>>>>>>>>>>> validating it? >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > It's strictly a test infrastructure change, it has no >>>>>>>>>>>>>> functional impact. For context, the changes included were from >>>>>>>>>>>>>> https://github.com/apache/beam/pull/25661 and >>>>>>>>>>>>>> https://github.com/apache/beam/pull/25654, both were keeping >>>>>>>>>>>>>> integration tests from running correctly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> > On Fri, Mar 3, 2023 at 2:09 PM Robert Bradshaw < >>>>>>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> The released artifacts seem to be missing the last commit >>>>>>>>>>>>>> at >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> https://github.com/apache/beam/commit/c528eab18b32342daed53b750fe330d30c7e5224 >>>>>>>>>>>>>> >> . Is this essential to the release, or just useful for >>>>>>>>>>>>>> validating it? >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Fri, Mar 3, 2023 at 11:02 AM Danny McCormick >>>>>>>>>>>>>> >> <dannymccorm...@google.com> wrote: >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > Thanks for calling that out, and thanks for helping me >>>>>>>>>>>>>> fix it! We should be all set now >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > On Fri, Mar 3, 2023 at 1:38 PM Robert Bradshaw < >>>>>>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> It appears your public key is not published in >>>>>>>>>>>>>> >> >> https://dist.apache.org/repos/dist/release/beam/KEYS . >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> On Fri, Mar 3, 2023 at 8:33 AM Anand Inguva via dev < >>>>>>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > +1 (non-binding) >>>>>>>>>>>>>> >> >> > Tested python wordcount quick start >>>>>>>>>>>>>> https://beam.apache.org/get-started/quickstart-py/ on Direct >>>>>>>>>>>>>> Runner and Dataflow Runner. >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > Thanks! >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > On Fri, Mar 3, 2023 at 11:21 AM Bruno Volpato via dev >>>>>>>>>>>>>> <dev@beam.apache.org> wrote: >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> >> >> >> +1 (non-binding) >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> >> >> >> Tested with >>>>>>>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowTemplates >>>>>>>>>>>>>> (Java SDK 11, Dataflow runner). >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> >> >> >> Thanks Danny! >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> >> >> >> On Thu, Mar 2, 2023 at 5:16 PM Danny McCormick via >>>>>>>>>>>>>> dev <dev@beam.apache.org> wrote: >>>>>>>>>>>>>> >> >> >>> >>>>>>>>>>>>>> >> >> >>> Hi everyone, >>>>>>>>>>>>>> >> >> >>> Please review and vote on release candidate #1 for >>>>>>>>>>>>>> the version 2.46.0, as follows: [ ] +1, Approve the release [ ] >>>>>>>>>>>>>> -1, Do not >>>>>>>>>>>>>> approve the release (please provide specific comments) Reviewers >>>>>>>>>>>>>> are >>>>>>>>>>>>>> encouraged to test their own use cases with the release >>>>>>>>>>>>>> candidate, and vote >>>>>>>>>>>>>> +1 if no issues are found. The complete staging area is >>>>>>>>>>>>>> available for your >>>>>>>>>>>>>> review, which includes: * GitHub Release notes [1], * the >>>>>>>>>>>>>> official Apache >>>>>>>>>>>>>> source release to be deployed to dist.apache.org [2], which >>>>>>>>>>>>>> is signed with the key with fingerprint >>>>>>>>>>>>>> FC383FCDE7D7E86699954EF2509872C8031C4DFB [3], * all artifacts to >>>>>>>>>>>>>> be >>>>>>>>>>>>>> deployed to the Maven Central Repository [4], * source code tag >>>>>>>>>>>>>> "v2.46.0-RC1" [5], * website pull request listing the release >>>>>>>>>>>>>> [6], the blog >>>>>>>>>>>>>> post [6], and publishing the API reference manual [7]. * Java >>>>>>>>>>>>>> artifacts >>>>>>>>>>>>>> were built with Gradle GRADLE_VERSION and OpenJDK/Oracle JDK >>>>>>>>>>>>>> JDK_VERSION. * >>>>>>>>>>>>>> Python artifacts are deployed along with the source release to >>>>>>>>>>>>>> the >>>>>>>>>>>>>> dist.apache.org [2] and PyPI[8]. * Go artifacts and >>>>>>>>>>>>>> documentation are available at pkg.go.dev [9] * Validation >>>>>>>>>>>>>> sheet with a tab for 2.46.0 release to help with validation >>>>>>>>>>>>>> [10]. * Docker >>>>>>>>>>>>>> images published to Docker Hub [11]. >>>>>>>>>>>>>> >> >> >>> * PR to run tests against release branch [12]. The >>>>>>>>>>>>>> vote will be open for at least 72 hours. It is adopted by >>>>>>>>>>>>>> majority >>>>>>>>>>>>>> approval, with at least 3 PMC affirmative votes. For guidelines >>>>>>>>>>>>>> on how to >>>>>>>>>>>>>> try the release in your projects, check out our blog post at >>>>>>>>>>>>>> /blog/validate-beam-release/. Thanks, Danny [1] >>>>>>>>>>>>>> https://github.com/apache/beam/milestone/9 [2] >>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.46.0/ [3] >>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/beam/KEYS [4] >>>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1306/ >>>>>>>>>>>>>> [5] https://github.com/apache/beam/tree/v2.46.0-RC1 [6] >>>>>>>>>>>>>> https://github.com/apache/beam/pull/25693 [7] >>>>>>>>>>>>>> https://github.com/apache/beam-site/pull/641 [8] >>>>>>>>>>>>>> https://pypi.org/project/apache-beam/2.46.0rc1/ [9] >>>>>>>>>>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.46.0-RC1/go/pkg/beam >>>>>>>>>>>>>> [10] >>>>>>>>>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=247587190 >>>>>>>>>>>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image >>>>>>>>>>>>>> >> >> >>> [12] https://github.com/apache/beam/pull/25600 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>
