Unless there are objections. I am proposing the option-1 short-term fix to unblock the 2.78 release cut. I will update the design doc for option-3, once 2.78 is done, it also needs alignment from Runner.
Thank you, Tarun. On Wed, Oct 28, 2026 at 4:45 PM Chamikara Jayalath <[email protected]> wrote: > > > On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <[email protected]> > wrote: > >> Hi, >> >> ` I understand there is some urgency to have a quick fix as multi-env >> pipelines might be blocked. If those pipelines are blocked already, do we >> have a workaround for affected users, for example to use a custom container >> instead of passing --requirements_file that collides in Python vs YAML >> environment?` >> >> > I do not have a good workaround for these customers. Multi-env YAML >> pipelines will fail. I am surprised there is no report, perhaps because the >> regression started only in 2.76 and multi-env is not a regular use case. >> > > This will include all Beam YAML users that use Python providers. > > https://beam.apache.org/documentation/sdks/yaml-providers/#python > > >> >> Thank you, >> Tarun. >> >> On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <[email protected]> >> wrote: >> >>> Thanks, Tarun. >>> >>> In addition to boot.go, SDK is also looking for some portion of the >>> staged files, like pickled_main_session >>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> >>> (somehow >>> we are still using it after cloudpickle migration for new purposes), in >>> #1/#2 SDK would need to be aware that there is an assigned worker ID (and >>> it is not the same ID anymore as the ID within boot.go, due to sibling >>> SDK protocol >>> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> >>> ). >>> >>> We could do a targeted fix just for requirements.txt, which is even more >>> narrow and reversible. Though I think the proper way to solve this is >>> on the runner. I understand there is some urgency to have a quick fix >>> as multi-env pipelines might be blocked. If those pipelines are blocked >>> already, do we have a workaround for affected users, for example to use a >>> custom container instead of passing --requirements_file that collides in >>> Python vs YAML environment? >>> >>> If we have a workaround, I'd suggest to try to prototype a runner side >>> fix before rushing in the SDK change. >>> >>> I think fixing the contract on the runner is a cleaner direction (see #A >>> vs #B in my earlier email), though others are welcome to weigh in here. >>> >>> If there are reasons to believe #B is the way to go (SDK must >>> deduplicate), then I'd explore if we can strengthen some guarantees for >>> files supplied via --files_to_stage (perhaps via a proto change), so that >>> we determine the location for these files in a more deterministic manner, >>> possilbly irrespective of the runner. >>> >>
