It looks like there is an experiment that can be used to temporarily override the sha checks while service side mitigation is in progress. This is not ideal, and we still have a race condition during artifact materialization in the latest SDK release. But this race is a preexisting problem. Therefore my preference would be not to incur technical debt in the SDK with a rushed workaround if we can avoid it, so I am voting -0 on implementing option 1 in the ongoing release.
I think the existing workaround (opt-out experiment for hash validations) can be acceptable until the next release since the experiment can be user-controlled, and this is probably safer than landing a change to artifact staging path done right before the release (bigger blast radius is something goes wrong). If we find new information/opinion/evidence that resolving staging conflict must be done in SDK rather than runner, then we can create a better mechanism (e.g. option 1 or variation thereof) for the next release. Thanks, Valentyn On Thu, Oct 29, 2026 at 3:22 PM Chamikara Jayalath <[email protected]> wrote: > > > On Thu, Oct 29, 2026 at 10:09 AM Tarun Annapareddy < > [email protected]> wrote: > >> Unless there are objections. I am proposing the option-1 short-term fix >> to unblock the 2.78 release cut. I will update the design doc for option-3, >> once 2.78 is done, it also needs alignment from Runner. >> > > Using option 1 to unblock the release sounds good to me. We can look into > options 2 or 3 as the long term fix subsequently. > > >> >> Thank you, >> Tarun. >> >> On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <[email protected]> >> wrote: >> >>> >>> >>> On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> ` I understand there is some urgency to have a quick fix as multi-env >>>> pipelines might be blocked. If those pipelines are blocked already, do we >>>> have a workaround for affected users, for example to use a custom container >>>> instead of passing --requirements_file that collides in Python vs YAML >>>> environment?` >>>> >>>> > I do not have a good workaround for these customers. Multi-env YAML >>>> pipelines will fail. I am surprised there is no report, perhaps because the >>>> regression started only in 2.76 and multi-env is not a regular use case. >>>> >>> >>> This will include all Beam YAML users that use Python providers. >>> >>> https://beam.apache.org/documentation/sdks/yaml-providers/#python >>> >>> >>>> >>>> Thank you, >>>> Tarun. >>>> >>>> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev < >>>> [email protected]> wrote: >>>> >>>>> Thanks, Tarun. >>>>> >>>>> In addition to boot.go, SDK is also looking for some portion of the >>>>> staged files, like pickled_main_session >>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> >>>>> (somehow >>>>> we are still using it after cloudpickle migration for new purposes), in >>>>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and >>>>> it is not the same ID anymore as the ID within boot.go, due to sibling >>>>> SDK protocol >>>>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> >>>>> ). >>>>> >>>>> We could do a targeted fix just for requirements.txt, which is even >>>>> more narrow and reversible. Though I think the proper way to solve >>>>> this is on the runner. I understand there is some urgency to have a >>>>> quick fix as multi-env pipelines might be blocked. If those pipelines are >>>>> blocked already, do we have a workaround for affected users, for example >>>>> to >>>>> use a custom container instead of passing --requirements_file that >>>>> collides >>>>> in Python vs YAML environment? >>>>> >>>>> If we have a workaround, I'd suggest to try to prototype a runner side >>>>> fix before rushing in the SDK change. >>>>> >>>>> I think fixing the contract on the runner is a cleaner direction (see >>>>> #A vs #B in my earlier email), though others are welcome to weigh in here. >>>>> >>>>> If there are reasons to believe #B is the way to go (SDK must >>>>> deduplicate), then I'd explore if we can strengthen some guarantees for >>>>> files supplied via --files_to_stage (perhaps via a proto change), so that >>>>> we determine the location for these files in a more deterministic manner, >>>>> possilbly irrespective of the runner. >>>>> >>>>
