+1 I agree Runner fix look like a good longer term fix. It addresses all SDK's(Java/Python/Go)
On Wed, Oct 28, 2026 at 4:34 PM Valentyn Tymofieiev <[email protected]> wrote: > > but it will increase worker memory, as each container will now download > and store dependencies separately > > I think with your original proposal to download files into > semi_persist_dir/worker_id, we will still consume extra disk space since > each SDK container will be downloading the dependencies separately. > > If we intentionally make multiple environments share staged files on the > host VM, we need to fix a race that can happen between writes and reads in > different environments: https://github.com/apache/beam/issues/28605 (was > autoclosed, but it may be relevant again for Python + YAML / XLANG > environments). This is another reason to consider different host > directories for different containers. > > On Wed, Oct 28, 2026 at 4:26 PM Valentyn Tymofieiev <[email protected]> > wrote: > >> Thanks, Tarun. >> >> In addition to boot.go, SDK is also looking for some portion of the >> staged files, like pickled_main_session >> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L405> >> (somehow >> we are still using it after cloudpickle migration for new purposes), in >> #1/#2 SDK would need to be aware that there is an assigned worker ID (and >> it is not the same ID anymore as the ID within boot.go, due to sibling >> SDK protocol >> <https://github.com/apache/beam/blob/0c5adeaff12d46edffa88adbb43fd4ea8fad0f26/sdks/python/container/boot.go#L278> >> ). >> >> We could do a targeted fix just for requirements.txt, which is even more >> narrow and reversible. Though I think the proper way to solve this is >> on the runner. I understand there is some urgency to have a quick fix as >> multi-env pipelines might be blocked. If those pipelines are blocked >> already, do we have a workaround for affected users, for example to use a >> custom container instead of passing --requirements_file that collides in >> Python vs YAML environment? >> >> If we have a workaround, I'd suggest to try to prototype a runner side >> fix before rushing in the SDK change. >> >> I think fixing the contract on the runner is a cleaner direction (see #A >> vs #B in my earlier email), though others are welcome to weigh in here. >> >> If there are reasons to believe #B is the way to go (SDK must >> deduplicate), then I'd explore if we can strengthen some guarantees for >> files supplied via --files_to_stage (perhaps via a proto change), so that >> we determine the location for these files in a more deterministic manner, >> possilbly irrespective of the runner. >> >
