Hi folks, I'm writing to share some thoughts and changes we would like to propose, in order to make *KubernetesExecutor*'s *multi_namespace_mode* more flexible & truly enterprise-ready.
*# Background & Issue* If you check *KubernetesExecutor*'s *multi_namespace_mode *configuration <https://github.com/apache/airflow/blob/2.4.3/airflow/config_templates/default_airflow.cfg#L1186-L1187>, you will notice the statement "*Will require creating a cluster-role for the scheduler*". That's because under the hood KubernetesExecutor is using *list_pod_for_all_namespaces() <https://github.com/apache/airflow/blob/2.4.3/airflow/executors/kubernetes_executor.py#L140-L142>* when it is *multi_namespace_mode*. This is a cluster-wise query and would surely need a cluster role. This would become a problem in some of the enterprise environments. For example: - The Kubernetes clusters may be managed by a central team. Each user only gets full access to namespace(s), and is never allowed to have a cluster-role. - A Kuberntes cluster may be shared by multiple teams, and it's never a good idea to grant anyone cluster-role. This does block some users from using Airflow's *KubernetesExecutor* in the vanilla shape, and these users have to make certain customisations (like what our team has to do). *# Proposal* We would like to propose refactoring KubernetesExecutor so that users can provide a list of namespaces in the configuration, and then Scheduler/Executor only needs to have a namespace-role to these namespaces. We already have a draft ready to be shared really soon. We would like to hear feedback & thoughts from the community about this idea, before we proceed to create a PR. Looking forward to your thoughts, and wish everyone a good weekend (and for some folks a nice Thanksgiving break) ahead! Regards, XD
