thanks all, it seemed to work but now i am getting a different problem, having issues in building pyarrow...
Step #0 - "build-shareloader-template": Step #4 - "dftester-image": <string>:36: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html Step #0 - "build-shareloader-template": Step #4 - "dftester-image": WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": Traceback (most recent call last): Step #0 - "build-shareloader-template": Step #4 - "dftester-image": File "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject Step #0 - "build-shareloader-template": Step #4 - "dftester-image": section = defn.get("tool", {})[tool_name] Step #0 - "build-shareloader-template": Step #4 - "dftester-image": ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ Step #0 - "build-shareloader-template": Step #4 - "dftester-image": KeyError: 'setuptools_scm' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": running bdist_wheel It is somehow getting messed up with a toml ? Could anyone advise? thanks Marco On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user <[email protected]> wrote: > > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies > is a great example. > > On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < > [email protected]> wrote: > >> In this case the Python version will be defined by the Python version >> installed in the docker image of your flex template. So, you'd have to >> build your flex template from a base image with Python 3.11. >> >> On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World <[email protected]> >> wrote: >> >>> Hello >>> no i am running my pipelien on GCP directly via a flex template, >>> configured using a Docker file >>> Any chances to do something in the Dockerfile to force the version at >>> runtime? >>> Thanks >>> >>> On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < >>> [email protected]> wrote: >>> >>>> Hello, >>>> >>>> Are you running your pipeline from the python 3.11 environment? If you >>>> are running from a python 3.11 environment and don't use a custom docker >>>> container image, DataflowRunner(Assuming Apache Beam on GCP means Apache >>>> Beam on DataflowRunner), will use Python 3.11. >>>> >>>> Thanks, >>>> Anand >>>> >>>
