[Question] Multiple images per pipeline

2023-09-05 Thread Kai Wohlfahrt
Hi all, I would like to construct a Beam pipeline with Python transforms, where different transforms use different container images, is this possible? Browsing through the design docs for cross-language transforms, it seems like it should in theory be possible, as each step can spawn it's own cont

Re: [Question] Multiple images per pipeline

2023-09-05 Thread Chamikara Jayalath via user
It's possible in theory but currently we don't have a good API for replacing the environment of a given transform when defining a pipeline. Environments are configured during transform expansion and if transforms use expansion services (with different dependencies) they will get unique environments