alexsanderp opened a new issue, #42831: URL: https://github.com/apache/airflow/issues/42831
### Apache Airflow version 2.10.2 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Scheduler Exception: ``` /home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py:765 FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0. /home/airflow/.local/lib/python3.11/site-packages/airflow/metrics/validators.py:95 DeprecationWarning: The statsd_allow_list option in [metrics] has been renamed to metrics_allow_list - the old setting has been used, but please update your config. /home/airflow/.local/lib/python3.11/site-packages/airflow/metrics/statsd_logger.py:184 RemovedInAirflow3Warning: The basic metric validator will be deprecated in the future in favor of pattern-matching. You can try this now by setting config option metrics_use_pattern_match to True. DataHurb2LineageBackend is not available. Please install the required dependencies. ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2024-10-08T15:26:01.247+0000] {_client.py:1026} INFO - HTTP Request: GET https://apacheairflow.gateway.scarf.sh/scheduler?version=2.10.2&python_version=3.11&platform=Linux&arch=x86_64&database=postgresql&db_version=12.20&executor=KubernetesExecutor "HTTP/1.1 200 OK" /home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:143 FutureWarning: The config section [kubernetes] has been renamed to [kubernetes_executor]. Please update your `conf.get*` call to use the new name [2024-10-08T15:26:01.361+0000] {executor_loader.py:254} INFO - Loaded executor: KubernetesExecutor [2024-10-08T15:26:01.437+0000] {scheduler_job_runner.py:935} INFO - Starting the scheduler [2024-10-08T15:26:01.437+0000] {scheduler_job_runner.py:942} INFO - Processing each file at most -1 times [2024-10-08T15:26:01.438+0000] {kubernetes_executor.py:287} INFO - Start Kubernetes executor [2024-10-08T15:26:01.471+0000] {kubernetes_executor_utils.py:140} INFO - Event: and now my watch begins starting at resource_version: 0 [2024-10-08T15:26:01.491+0000] {kubernetes_executor.py:208} INFO - Found 0 queued task instances [2024-10-08T15:26:01.493+0000] {scheduler_job_runner.py:1847} INFO - Adopting or resetting orphaned tasks for active dag runs [2024-10-08T15:26:01.653+0000] {scheduler_job_runner.py:1001} ERROR - Exception when executing SchedulerJob._run_scheduler_loop Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 984, in _execute self._run_scheduler_loop() File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1067, in _run_scheduler_loop self.adopt_or_reset_orphaned_tasks() File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py", line 97, in wrapper return func(*args, session=session, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1850, in adopt_or_reset_orphaned_tasks for attempt in run_with_db_retries(logger=self.log): File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 443, in __iter__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda> self._add_action_func(lambda rs: rs.outcome.result()) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1894, in adopt_or_reset_orphaned_tasks to_reset.extend(executor.try_adopt_task_instances(tis)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 571, in try_adopt_task_instances pod_list = self._list_pods(query_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 162, in _list_pods pods.extend(dynamic_client.get(resource=pod_resource, namespace=namespace, **query_kwargs).items) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 62, in inner return serializer(self, json.loads(resp.data.decode('utf8'))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/resource.py", line 294, in __init__ for item in instance['items']: TypeError: 'NoneType' object is not iterable [2024-10-08T15:26:01.656+0000] {kubernetes_executor.py:752} INFO - Shutting down Kubernetes executor [2024-10-08T15:26:01.700+0000] {scheduler_job_runner.py:1014} INFO - Exited execute loop Traceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/__main__.py", line 62, in main args.func(args) File "/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py", line 115, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 59, in scheduler run_command_with_daemon_option( File "/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option callback() File "/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in <lambda> callback=lambda: _run_scheduler_job(args), ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 48, in _run_scheduler_job run_job(job=job_runner.job, execute_callable=job_runner._execute) File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py", line 97, in wrapper return func(*args, session=session, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/job.py", line 421, in run_job return execute_job(job, execute_callable=execute_callable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/job.py", line 450, in execute_job ret = execute_callable() ^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 984, in _execute self._run_scheduler_loop() File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1067, in _run_scheduler_loop self.adopt_or_reset_orphaned_tasks() File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py", line 97, in wrapper return func(*args, session=session, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1850, in adopt_or_reset_orphaned_tasks for attempt in run_with_db_retries(logger=self.log): File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 443, in __iter__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda> self._add_action_func(lambda rs: rs.outcome.result()) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1894, in adopt_or_reset_orphaned_tasks to_reset.extend(executor.try_adopt_task_instances(tis)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 571, in try_adopt_task_instances pod_list = self._list_pods(query_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 162, in _list_pods pods.extend(dynamic_client.get(resource=pod_resource, namespace=namespace, **query_kwargs).items) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 62, in inner return serializer(self, json.loads(resp.data.decode('utf8'))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/resource.py", line 294, in __init__ for item in instance['items']: TypeError: 'NoneType' object is not iterable ``` ### What you think should happen instead? Scheduler running ### How to reproduce I don't know how I reproduced it, I just started using version 2.10.2 and this error has been occurring ### Operating System Kubernetes GKE ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==8.28.0 apache-airflow-providers-celery==3.8.1 apache-airflow-providers-cncf-kubernetes==8.4.1 apache-airflow-providers-common-compat==1.2.0 apache-airflow-providers-common-io==1.4.0 apache-airflow-providers-common-sql==1.16.0 apache-airflow-providers-docker==3.13.0 apache-airflow-providers-elasticsearch==5.5.0 apache-airflow-providers-fab==1.3.0 apache-airflow-providers-ftp==3.11.0 apache-airflow-providers-google==10.19.0 apache-airflow-providers-grpc==3.6.0 apache-airflow-providers-hashicorp==3.8.0 apache-airflow-providers-http==4.13.0 apache-airflow-providers-imap==3.7.0 apache-airflow-providers-microsoft-azure==10.4.0 apache-airflow-providers-mysql==5.7.0 apache-airflow-providers-odbc==4.7.0 apache-airflow-providers-openlineage==1.11.0 apache-airflow-providers-postgres==5.12.0 apache-airflow-providers-redis==3.8.0 apache-airflow-providers-sendgrid==3.6.0 apache-airflow-providers-sftp==4.11.0 apache-airflow-providers-slack==8.9.0 apache-airflow-providers-smtp==1.8.0 apache-airflow-providers-snowflake==5.7.0 apache-airflow-providers-sqlite==3.9.0 apache-airflow-providers-ssh==3.13.1 ### Deployment Official Apache Airflow Helm Chart ### Deployment details Helm Chart + ArgoCD ### Anything else? This issue is crashing our production environment ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org