TLDR I am getting inconsistent but frequent connection errors when using google.cloud modules.
SUMMARY I am repeatedly seeing `AttributeError: 'ConnectionError' object has no attribute 'message'` when using the `google.cloud` collection. This error has emerged in the last week or so, in a script that has not been modified and was previously working fine. It frequently is triggered by starting/stopping instances using gcp_compute_instance module <https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_instance_module.html#ansible-collections-google-cloud-gcp-compute-instance-module>, but has also occurred when creating a disk using gcp_compute_disk module <https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_disk_module.html#ansible-collections-google-cloud-gcp-compute-disk-module> . For context, my Ansible scripts are interfacing with GCP resources to loop test a software upgrade, but I can't get any further than 1x success without this error occurring. It is inconsistent and and has so far happened at many points during the software upgrade process, whenever `google.cloud` modules are used. The tasks do actually complete their task: the GCP resources are either started, stopped or created. But the returned error ends the play. Steps taken to investigate so far: - Ran with v*4 verbosity to review in greater detail. - Confirmed there have been no modifications to the Ansible scripts since the last successful loop tests. - Pinged 'compute.googleapis.com' from the host VM and confirmed it responded. - Checked GCP logs for affected instances and the used service account. No errors were reported. - Changed the host machine and network. - Checked for similar issues/PR's under the Ansible and google.cloud repositories. Any help is very appreciated, and please let me know if I can clarify anything. ISSUE TYPE - Bug Report COMPONENT NAME `google.cloud.compute_instance` `google.cloud.disk_module` ANSIBLE VERSION ```paste below ansible [core 2.16.4] config file = /tmp/ien-iac/configure/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3) jinja version = 3.1.3 libyaml = True ``` COLLECTION VERSION ``` # /root/.ansible/collections/ansible_collections Collection Version ------------ ------- google.cloud 1.3.0 # /usr/local/lib/python3.10/dist-packages/ansible_collections Collection Version ------------ ------- google.cloud 1.3.0 ``` CONFIGURATION ``` CALLBACKS_ENABLED(/tmp/ien-iac/configure/ansible.cfg) = ['timer'] CONFIG_FILE() = /tmp/ien-iac/configure/ansible.cfg DEFAULT_HOST_LIST(/tmp/ien-iac/configure/ansible.cfg) = ['/tmp/ien-iac/configure/dsds.gcp.yml'] DEFAULT_LOAD_CALLBACK_PLUGINS(/tmp/ien-iac/configure/ansible.cfg) = True DEFAULT_REMOTE_USER(/tmp/ien-iac/configure/ansible.cfg) = installer DEFAULT_STDOUT_CALLBACK(/tmp/ien-iac/configure/ansible.cfg) = yaml DEFAULT_TIMEOUT(/tmp/ien-iac/configure/ansible.cfg) = 30 DEFAULT_TRANSPORT(/tmp/ien-iac/configure/ansible.cfg) = ssh DEFAULT_VAULT_PASSWORD_FILE(/tmp/ien-iac/configure/ansible.cfg) = /tmp/ien-iac/configure/vault-pass.txt DISPLAY_SKIPPED_HOSTS(/tmp/ien-iac/configure/ansible.cfg) = False HOST_KEY_CHECKING(/tmp/ien-iac/configure/ansible.cfg) = False INTERPRETER_PYTHON(/tmp/ien-iac/configure/ansible.cfg) = /usr/bin/python3 INVENTORY_ENABLED(/tmp/ien-iac/configure/ansible.cfg) = ['gcp_compute'] PERSISTENT_COMMAND_TIMEOUT(/tmp/ien-iac/configure/ansible.cfg) = 660 ``` OS / ENVIRONMENT I am running the commands in a minimised Ubuntu Docker container which is used for a consistent CI tooled environment at my workplace. There are many other Ansible roles in this environment that do not use the `google.cloud` collection and aren't experiencing issues at the moment. ``` **OS** Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy **CPU** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 40 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: QEMU Virtual CPU version 2.5+ CPU family: 15 Model: 107 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: 1 BogoMIPS: 4389.68 Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xto pology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm cpuid_fault pti **NETWORK** *-network description: Ethernet controller product: Virtio network device vendor: Red Hat, Inc. physical id: 12 bus info: pci@0000:00:12.0 version: 00 width: 64 bits clock: 33MHz capabilities: msix bus_master cap_list rom configuration: driver=virtio-pci latency=0 resources: irq:10 ioport:f0a0(size=32) memory:fea53000-fea53fff memory:fd608000-fd60bfff memory:fea00000-fea3ffff ... ``` ANSIBLE SCRIPTS ```yaml --- # startup_node.yml - name: Query for GCP instance info ansible.builtin.include_tasks: "{{ rpath }}/validate_node.yml" - name: Startup the node when: node_status != "RUNNING" google.cloud.gcp_compute_instance: zone: "{{ zone }}" name: "{{ node }}" project: "{{ project }}" auth_kind: "{{ auth_kind }}" service_account_file: "{{ service_account_file }}" status: RUNNING deletion_protection: false machine_type: "{{ machine_type }}" labels: "{{ node_labels }}" --- # shutdown_node.yml - name: Query for GCP instance info ansible.builtin.include_tasks: "{{ rpath }}/validate_node.yml" - name: Shutdown the node when: node_status != "TERMINATED" google.cloud.gcp_compute_instance: zone: "{{ zone }}" name: "{{ node }}" project: "{{ project }}" auth_kind: "{{ auth_kind }}" service_account_file: "{{ service_account_file }}" status: TERMINATED deletion_protection: false machine_type: "{{ machine_type }}" labels: "{{ node_labels }}" --- # create_disk.yml - name: Display address of source snapshot for new disk ansible.builtin.debug: msg: "{{ self_link | community.general.dict_kv('selfLink') }}" - name: Set disk size as fact ansible.builtin.set_fact: disk_size: 100 - name: Create disk - "{{ disk }}" google.cloud.gcp_compute_disk: name: "{{ disk }}" size_gb: "{{ disk_size }}" zone: "{{ zone }}" project: "{{ project }}" auth_kind: "{{ auth_kind }}" service_account_file: "{{ service_account_file }}" source_snapshot: "{{ self_link | community.general.dict_kv('selfLink') }}" type: "pd-ssd" # Module expands this into the full URL using the project and zone fields. state: present --- # validate_node.yml - name: Display node name ansible.builtin.debug: var: node - name: Query gcp for node info google.cloud.gcp_compute_instance_info: zone: "{{ zone }}" filters: - name = "{{ node }}" project: "{{ project }}" auth_kind: "{{ auth_kind }}" service_account_file: "{{ service_account_file }}" register: node_info - name: Error catching ansible.builtin.fail: msg: "Node not found." when: node_info.resources == [] - ansible.builtin.set_fact: node_labels: "{{ node_info.resources[0].labels | community.general.dict }}" machine_type: "{{ node_info.resources[0].machineType }}" node_status: "{{ node_info.resources[0].status }}" ``` EXPECTED RESULTS Instances should start and stop. Disks should be created. These modules shouldn't return errors while appearing to succeed within GCP. ACTUAL RESULTS In the example below the Ansible play attempted to stop an instance. The instance was running at the time. This task returned `OSError: [Network is unreachable]` and ended the play, but GCP logs show the API request and the instance was successfully stopped. ```paste below 2024-04-11 14:04:18,512 p=997529 u=root n=ansible | TASK [ien-upgrade : Shutdown the node] ***************************************** 2024-04-11 14:04:18,512 p=997529 u=root n=ansible | The full traceback is: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 616, in connect self.sock = sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 213, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='compute.googleapis.com', port=443): Max retries exceeded with url: /compute/v1/projects/ret-edge-dsds-superbad/zones/europe-west2-c/operations/operation-1712844202693-615d29e41d288-5d3a5ea2-94699733 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 150, in full_get File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/google/auth/transport/requests.py", line 541, in request response = super(AuthorizedSession, self).request( File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='compute.googleapis.com', port=443): Max retries exceeded with url: /compute/v1/projects/ret-edge-dsds-superbad/zones/europe-west2-c/operations/operation-1712844202693-615d29e41d288-5d3a5ea2-94699733 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 107, in <module> _ansiballz_main() File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 99, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 47, in invoke_module runpy.run_module(mod_name='ansible_collections.google.cloud.plugins.modules.gcp_compute_instance', init_globals=dict(_module_fqn='ansible_collections.google.cloud.plugins.modules.gcp_compute_instance', _modlib_path=modlib_path), File "/usr/lib/python3.10/runpy.py", line 224, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1905, in <module> File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1211, in main File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1491, in run File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1501, in stop File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1407, in wait_for_operation File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1421, in wait_for_completion File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1293, in fetch_resource File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 85, in get File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 153, in full_get AttributeError: 'ConnectionError' object has no attribute 'message' 2024-04-11 14:04:18,516 p=997529 u=root n=ansible | fatal: [localhost]: FAILED! => changed=false module_stderr: |- Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 616, in connect self.sock = sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 213, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='compute.googleapis.com', port=443): Max retries exceeded with url: /compute/v1/projects/ret-edge-dsds-superbad/zones/europe-west2-c/operations/operation-1712844202693-615d29e41d288-5d3a5ea2-94699733 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 150, in full_get File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/google/auth/transport/requests.py", line 541, in request response = super(AuthorizedSession, self).request( File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='compute.googleapis.com', port=443): Max retries exceeded with url: /compute/v1/projects/ret-edge-dsds-superbad/zones/europe-west2-c/operations/operation-1712844202693-615d29e41d288-5d3a5ea2-94699733 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x706c27a50670>: Failed to establish a new connection: [Errno 101] Network is unreachable')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 107, in <module> _ansiballz_main() File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 99, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "/root/.ansible/tmp/ansible-tmp-1712844199.9392037-998908-265836207911416/AnsiballZ_gcp_compute_instance.py", line 47, in invoke_module runpy.run_module(mod_name='ansible_collections.google.cloud.plugins.modules.gcp_compute_instance', init_globals=dict(_module_fqn='ansible_collections.google.cloud.plugins.modules.gcp_compute_instance', _modlib_path=modlib_path), File "/usr/lib/python3.10/runpy.py", line 224, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1905, in <module> File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1211, in main File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1491, in run File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1501, in stop File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1407, in wait_for_operation File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1421, in wait_for_completion File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/modules/gcp_compute_instance.py", line 1293, in fetch_resource File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 85, in get File "/tmp/ansible_google.cloud.gcp_compute_instance_payload_cvk7rrr8/ansible_google.cloud.gcp_compute_instance_payload.zip/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 153, in full_get AttributeError: 'ConnectionError' object has no attribute 'message' module_stdout: '' msg: |- MODULE FAILURE See stdout/stderr for the exact error rc: 1 ``` -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/36d92a7f-8c6c-46da-8853-dfdbc757f445n%40googlegroups.com.
