> but it will increase worker memory, as each container will now download and store dependencies separately
I think with your original proposal to download files into semi_persist_dir/worker_id, we will still consume extra disk space since each SDK container will be downloading the dependencies separately. If we intentionally make multiple environments share staged files on the host VM, we need to fix a race that can happen between writes and reads in different environments: https://github.com/apache/beam/issues/28605 (was autoclosed, but it may be relevant again for Python + YAML / XLANG environments). This is another reason to consider different host directories for different containers. On Wed, Oct 28, 2026 at 4:26 PM Valentyn Tymofieiev <[email protected]> wrote: > Thanks, Tarun. > > In addition to boot.go, SDK is also looking for some portion of the staged > files, like pickled_main_session > <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> > (somehow > we are still using it after cloudpickle migration for new purposes), in > #1/#2 SDK would need to be aware that there is an assigned worker ID (and > it is not the same ID anymore as the ID within boot.go, due to sibling > SDK protocol > <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> > ). > > We could do a targeted fix just for requirements.txt, which is even more > narrow and reversible. Though I think the proper way to solve this is on > the runner. I understand there is some urgency to have a quick fix as > multi-env pipelines might be blocked. If those pipelines are blocked > already, do we have a workaround for affected users, for example to use a > custom container instead of passing --requirements_file that collides in > Python vs YAML environment? > > If we have a workaround, I'd suggest to try to prototype a runner side fix > before rushing in the SDK change. > > I think fixing the contract on the runner is a cleaner direction (see #A > vs #B in my earlier email), though others are welcome to weigh in here. > > If there are reasons to believe #B is the way to go (SDK must > deduplicate), then I'd explore if we can strengthen some guarantees for > files supplied via --files_to_stage (perhaps via a proto change), so that > we determine the location for these files in a more deterministic manner, > possilbly irrespective of the runner. >
