I think the problem is you do not specify sdk_container_image when running your template.
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template has more details. Basically, you do not need to https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile for your template launcher. You can use the same image for both launcher and dataflow workers. You only need to copy python_template_launcher to your image like https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. Then you can use this image for both launcher and dataflow workers. When running the template job, you need to add --parameters sdk_container_image=$SDK_CONTAINER_IMAGE. On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <paulhtremb...@gmail.com> wrote: > A few weeks ago I had posted a problem I had with importing the Google > Cloud Secret Manager library in Python. > > Here is the problem and solution: > > > https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager > > -- > Henry Tremblay > Data Engineer, Best Buy >
