hey everyone... i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates.
the TL;DR is that we've been trying to update the k8s client libs to something much more modern. however, for us to do this, we need to update our very old k8s and minikube versions. the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions. i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins). for this to happen, two PRs will need to be merged... one for 2.4.1 and one for master. is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release? this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously. both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why). here's the primary jira we're using to track this upgrade: https://issues.apache.org/jira/browse/SPARK-26742 thanks in advance, shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
