jason810496 opened a new issue, #62769: URL: https://github.com/apache/airflow/issues/62769
### Body ## What The test sometimes fails. For example: https://github.com/apache/airflow/actions/runs/22609497337/job/65509753048?pr=62758 ## Traceback Logs summary: ```python ======================================================================================== short test summary info ========================================================================================= FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes - AssertionError: Span name 'task2' wasn't found in children span names. It's not a child of span 'otel_test_dag_with_pause_between_tasks'. FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_exits_gracefully_in_the_middle_of_the_first_task - AssertionError: Span name 'task1_sub_span3' wasn't found in children span names. It's not a child of span 'task1_sub_span2'. ===================================================================== 2 failed, 7 passed, 10 skipped, 1 warning in 580.16s (0:09:40) ===================================================================== ``` <details> <summary>full traceback:</summary> ```python Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552 Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics self._exporter.export( File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export resp = self._export(serialized_data, deadline_sec - time()) File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) Warning: ] Invalid type UUID for attribute 'airflow.task.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types [opentelemetry.attributes] loc=__init__.py:111 Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552 Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics self._exporter.export( File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export resp = self._export(serialized_data, deadline_sec - time()) File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552 Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics self._exporter.export( File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export resp = self._export(serialized_data, deadline_sec - time()) File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552 Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn sock = connection.create_connection( File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request conn.request( File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request self.endheaders() File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/python/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect self.sock = self._new_conn() File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn raise NameResolutionError(self.host, self, e) from e urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics self._exporter.export( File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export resp = self._export(serialized_data, deadline_sec - time()) File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export resp = self._session.post( File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) -- err-end ERROR integration.otel.otel_utils:otel_utils.py:146 Failed to parse JSON: { "resource_metrics": [ { "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.39.1", "service.name": "airflow" }, /root/airflow/logs/dag_id=otel_test_dag_with_pause_in_task/run_id=manual__2026-03-03T05:53:30.782643+00:00/task_id=task1/attempt=1.log File "/usr/python/lib/python3.10/logging/__init__.py", line 1100, in emit msg = self.format(record) File "/usr/python/lib/python3.10/logging/__init__.py", line 943, in format return fmt.format(record) File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 1154, in format ed = cast(EventDict, proc(logger, meth_name, ed)) File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 764, in __call__ if len(args) == 1 and isinstance(args[0], dict) and args[0]: File "/usr/python/bin/celery", line 10, in <module> sys.exit(main()) File "/usr/python/lib/python3.10/site-packages/celery/__main__.py", line 15, in main sys.exit(_main()) File "/usr/python/lib/python3.10/site-packages/celery/bin/celery.py", line 227, in main return celery(auto_envvar_prefix="CELERY") File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1485, in __call__ return self.main(*args, **kwargs) File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1406, in main rv = self.invoke(ctx) File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1873, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1269, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/python/lib/python3.10/site-packages/click/core.py", line 824, in invoke return callback(*args, **kwargs) File "/usr/python/lib/python3.10/site-packages/click/decorators.py", line 34, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/python/lib/python3.10/site-packages/celery/bin/base.py", line 135, in caller return f(ctx, *args, **kwargs) File "/usr/python/lib/python3.10/site-packages/celery/bin/worker.py", line 367, in worker worker.start() File "/usr/python/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start self.blueprint.start(self) File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start step.start(parent) File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start return self.obj.start() File "/usr/python/lib/python3.10/site-packages/celery/concurrency/base.py", line 130, in start self.on_start() File "/usr/python/lib/python3.10/site-packages/celery/concurrency/prefork.py", line 112, in on_start P = self._pool = Pool(processes=self.limit, File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 475, in __init__ super().__init__(processes, *args, **kwargs) File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1046, in __init__ self._create_worker_process(i) File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 493, in _create_worker_process return super()._create_worker_process(i) File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1158, in _create_worker_process w.start() File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 120, in start self._popen = self._Popen(self) File "/usr/python/lib/python3.10/site-packages/billiard/context.py", line 331, in _Popen return Popen(process_obj) File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 22, in __init__ self._launch(process_obj) File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 77, in _launch code = process_obj._bootstrap() File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 323, in _bootstrap self.run() File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 110, in run self._target(*self._args, **self._kwargs) File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 292, in __call__ sys.exit(self.workloop(pid=pid)) File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 362, in workloop result = (True, prepare_result(fun(*args, **kwargs))) File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 694, in fast_trace_task R, I, T, Rstr = tasks[task].__trace__( File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 479, in trace_task R = retval = fun(*args, **kwargs) File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 779, in __protected_call__ return self.run(*args, **kwargs) File "/opt/airflow/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload supervise( File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 2090, in supervise process = ActivitySubprocess.start( File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 1004, in start proc: Self = super().start(id=what.id, client=client, target=target, logger=logger, **kwargs) File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 514, in start _fork_main(child_requests, child_stdout, child_stderr, child_logs.fileno(), target) File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 413, in _fork_main target() File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main main() File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1825, in main state, _, error = run(ti, context, log) File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1236, in run result = _execute_task(context=context, ti=ti, log=log) File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1651, in _execute_task result = ctx.run(execute, context=context) File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper return func(self, *args, **kwargs) File "/opt/airflow/task-sdk/src/airflow/sdk/bases/decorator.py", line 360, in execute return_value = super().execute(context) File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper return func(self, *args, **kwargs) File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 232, in execute return_value = self.execute_callable() File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 255, in execute_callable return runner.run(*self.op_args, **self.op_kwargs) File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/callback_runner.py", line 97, in run return func(*args, **kwargs) File "/opt/airflow/airflow-core/tests/integration/otel/dags/otel_test_dag_with_pause_in_task.py", line 76, in task1 logger.info("Found ti.context_carrier: %s.", context_carrier) { "name": "task1_sub_span3", "context": { "trace_id": "0x186713f4ffcd23351586d08d9fd5ed34", "span_id": "0x57ffb0578ea05d12", "trace_state": "[]" }, "kind": "SpanKind.INTERNAL", "parent_id": "0x70dad54d80c553f6", "start_time": "2026-03-03T05:53:42.271939Z", "end_time": "2026-03-03T05:53:42.272071Z", "status": { "status_code": "UNSET" }, "attributes": { "attr3": "val3" }, "events": [], "links": [], "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.39.1", "service.name": "airflow" }, "schema_url": "" } } ``` </details> ### Committer - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
