On Wed, Oct 28, 2026 at 4:35 PM Tarun Annapareddy <[email protected]> wrote:
> Hi, > > ` I understand there is some urgency to have a quick fix as multi-env > pipelines might be blocked. If those pipelines are blocked already, do we > have a workaround for affected users, for example to use a custom container > instead of passing --requirements_file that collides in Python vs YAML > environment?` > > > I do not have a good workaround for these customers. Multi-env YAML > pipelines will fail. I am surprised there is no report, perhaps because the > regression started only in 2.76 and multi-env is not a regular use case. > This will include all Beam YAML users that use Python providers. https://beam.apache.org/documentation/sdks/yaml-providers/#python > > Thank you, > Tarun. > > On Wed, Oct 28, 2026 at 4:27 PM Valentyn Tymofieiev <[email protected]> > wrote: > >> Thanks, Tarun. >> >> In addition to boot.go, SDK is also looking for some portion of the >> staged files, like pickled_main_session >> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> >> (somehow >> we are still using it after cloudpickle migration for new purposes), in >> #1/#2 SDK would need to be aware that there is an assigned worker ID (and >> it is not the same ID anymore as the ID within boot.go, due to sibling >> SDK protocol >> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> >> ). >> >> We could do a targeted fix just for requirements.txt, which is even more >> narrow and reversible. Though I think the proper way to solve this is >> on the runner. I understand there is some urgency to have a quick fix as >> multi-env pipelines might be blocked. If those pipelines are blocked >> already, do we have a workaround for affected users, for example to use a >> custom container instead of passing --requirements_file that collides in >> Python vs YAML environment? >> >> If we have a workaround, I'd suggest to try to prototype a runner side >> fix before rushing in the SDK change. >> >> I think fixing the contract on the runner is a cleaner direction (see #A >> vs #B in my earlier email), though others are welcome to weigh in here. >> >> If there are reasons to believe #B is the way to go (SDK must >> deduplicate), then I'd explore if we can strengthen some guarantees for >> files supplied via --files_to_stage (perhaps via a proto change), so that >> we determine the location for these files in a more deterministic manner, >> possilbly irrespective of the runner. >> >
