It's possible in theory but currently we don't have a good API for replacing the environment of a given transform when defining a pipeline. Environments are configured during transform expansion and if transforms use expansion services (with different dependencies) they will get unique environments.
So a workaround might be to start a different expansion service (Python) [1] for each transform that needs a unique environment (if multiple transforms can use the same environment you can group them into the same expansion service) and use the cross-language APIs to attach such transforms with unique environment to the pipeline [2] but this can be cumbersome if you need to support a lot of environments. Thanks, Cham [1] https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/#advanced-start-an-expansion-service [2] https://beam.apache.org/documentation/programming-guide/#use-x-lang-transforms On Tue, Sep 5, 2023 at 12:56 AM Kai Wohlfahrt <k...@charmtx.com> wrote: > Hi all, > > I would like to construct a Beam pipeline with Python transforms, where > different transforms use different container images, is this possible? > Browsing through the design docs for cross-language transforms, it seems > like it should in theory be possible, as each step can spawn it's own > container if I understand correctly. > > Unfortunately, I'm not having any luck piecing together the concrete steps > necessary to construct such a pipeline. I was only able to find the > `--environment_config` flag, but this applies a single image to the whole > pipeline. Ideallly, this would work on the Flink runner, but I'm open to > setting up alternative runners if required for this. > > The motivation behind this request is that we'd like to build a pipeline > that chains together various bits of software for processing protein > structures. These packages are either poorly packaged or have many > dependencies (or both), making it difficult to construct a single > environment that contains all of the transitive dependencies without > conflicts. It would be much nicer to be able to construct a single image > per step, and then chain these together into a unified pipeline with Beam. > > Many thanks, > Kai >
