Hi, ` I understand there is some urgency to have a quick fix as multi-env pipelines might be blocked. If those pipelines are blocked already, do we have a workaround for affected users, for example to use a custom container instead of passing --requirements_file that collides in Python vs YAML environment?`
> I do not have a good workaround for these customers. Multi-env YAML pipelines will fail. I am surprised there is no report, perhaps because the regression started only in 2.76 and multi-env is not a regular use case. Thank you, Tarun. On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <[email protected]> wrote: > Thanks, Tarun. > > In addition to boot.go, SDK is also looking for some portion of the staged > files, like pickled_main_session > <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> > (somehow > we are still using it after cloudpickle migration for new purposes), in > #1/#2 SDK would need to be aware that there is an assigned worker ID (and > it is not the same ID anymore as the ID within boot.go, due to sibling > SDK protocol > <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> > ). > > We could do a targeted fix just for requirements.txt, which is even more > narrow and reversible. Though I think the proper way to solve this is on > the runner. I understand there is some urgency to have a quick fix as > multi-env pipelines might be blocked. If those pipelines are blocked > already, do we have a workaround for affected users, for example to use a > custom container instead of passing --requirements_file that collides in > Python vs YAML environment? > > If we have a workaround, I'd suggest to try to prototype a runner side fix > before rushing in the SDK change. > > I think fixing the contract on the runner is a cleaner direction (see #A > vs #B in my earlier email), though others are welcome to weigh in here. > > If there are reasons to believe #B is the way to go (SDK must > deduplicate), then I'd explore if we can strengthen some guarantees for > files supplied via --files_to_stage (perhaps via a proto change), so that > we determine the location for these files in a more deterministic manner, > possilbly irrespective of the runner. >
