You shouldn’t have to use an sdk_container_image. It is not in the docs, and I talked to Google, and they said a container image is not needed. Also, why does the requests library work, and the secret manager does not?
From: XQ Hu via user <[email protected]> Sent: Sunday, November 3, 2024 1:53 PM To: [email protected] Cc: XQ Hu <[email protected]> Subject: Re: Solution to import problem I think the problem is you do not specify sdk_container_image when running your template. https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template has more details. Basically, you do not need to https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile for your template launcher. You can use the same image for both launcher and dataflow workers. You only need to copy python_template_launcher to your image like https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. Then you can use this image for both launcher and dataflow workers. When running the template job, you need to add --parameters sdk_container_image=$SDK_CONTAINER_IMAGE. On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <[email protected]<mailto:[email protected]>> wrote: A few weeks ago I had posted a problem I had with importing the Google Cloud Secret Manager library in Python. Here is the problem and solution: https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager -- Henry Tremblay Data Engineer, Best Buy
