On Wed, Oct 28, 2026 at 2:13 PM Tarun Annapareddy <[email protected]> wrote:
> Hi everyone, > > New findings and discussions are available at go/beam-pr/40699 > <https://goto.google.com/beam-pr/40699>. While the high level idea is to > align Python dependency management similar to Java(previous work > <https://github.com/apache/beam/issues/20009>). We realized that some > Python use cases assumed a fixed path existed. For example -- > files_to_stage > <https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/>, > which is used by run_inference > <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116>. > With this proposal the SEMI_PERSISTENT_DIRECTORY is no longer a static > path, known during pipeline submission. So, this would break their > pipelines; they should apply a similar fix (go/beam-pr/40795 > <https://goto.google.com/beam-pr/40795>). > > Here are a few options: > > 1. [Quick Fix]: Narrow down the scope of problem. Instead of moving all > the dependencies into separate environment-based paths. We narrow > down(hardcode) specific files(example > <https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L71>, > which we already do) This will allow us to solve this issue in the latest > release cut (planned today). > 2. [SDK fix]: Instead of listing all these files manually in the boot.go. > Stager marks these files with a separate marker in the pipeline proto. The > boot process downloads the files into separate directories (shared vs. > environment-based) > 3.[Runner fix]: Runner can host different folder, mounted at the same path > in each container(host /var/opt/google/<container> mounted at > /var/opt/google) So, there is no conflict between the containers. May be > with this, we can remove environment-based downloads for Java as well. It > looks clean, but it will increase worker memory, as each container will now > download and store dependencies separately. > I would choose option 3 to unblock the release. For the long term fix, option 3 probably has the least disruption for existing customers who might be using the hard coded staging path for runners like Dataflow but this probably also the riskiest (changes the way containers are setup for all SDKs that use the runner). If we choose option 2 we can do a more limited fix just to Python SDK (which is the only SDK affected) but in that case we should provide a clear util for users to easily find staged files locally on the workers. Thanks, Cham > > With all these options, customers using the run_inference > <https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L116> > pattern > are still un-impacted. I recommend option-1, to unblock multi-env YAML > pipelines, option-3 needs agreement on the memory trade-off. > > Thank you, > Tarun. > > > On Wed, Oct 21, 2026 at 11:14 AM Valentyn Tymofieiev <[email protected]> > wrote: > >> +1, thank you! >> >> On Wed, Oct 21, 2026 at 10:56 AM Chamikara Jayalath via dev < >> [email protected]> wrote: >> >>> +1. Thanks a lot for looking into this and providing a fix Tarun. >>> >>> - Cham >>> >>> On Wed, Oct 21, 2026 at 6:41 AM Kenneth Knowles <[email protected]> wrote: >>> >>>> Nice catch, and good write-up. Thanks for the clear summary in the doc, >>>> too. >>>> >>>> Kenn >>>> >>>> On Tue, Oct 20, 2026 at 11:02 PM Tarun Annapareddy via dev < >>>> [email protected]> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> Recently, we have encountered Dataflow Multi-Environment pipelines >>>>> crash-looping due to integrity check failures. However, while debugging we >>>>> realized it's a long-standing issue in Beam Python dependency staging. >>>>> This >>>>> fix is small, but a bit convoluted, so I have created a doc and attached >>>>> the PR. >>>>> >>>>> Doc: >>>>> https://docs.google.com/document/d/1gNfjbIYNzbv15-RjCj0ad4MHenVPoSjUEcH0Ok3iNyU/edit?usp=sharing >>>>> >>>>> PR: go/beam-pr/40699 <https://goto.google.com/beam-pr/40699> >>>>> >>>>> Thank you, >>>>> Tarun. >>>>> >>>>
