Re: Spark 3 pod template for the driver

If I could muster a guess, you still need to specify the executor image. As is, this will only specify the driver image. You can specify it as --conf spark.kubernetes.container.image or --conf spark.kubernetes.executor.container.image -- Sent from: http://apache-spark-developers-list.1001551.n3

Re: Getting the ball started on a 2.4.6 release

Yes, watching the pod yaml could work for this. Just gotta set up some kind of thing to do that, thanks for clueing me into that. And sounds great re: Spark 2.5. Having a transitional release makes sense I think. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

There's other information you can obtain from the Pod metadata on a describe than just from the logs, which are typically what's being printed by the Application itself. I've also found that Spark has some trouble obtaining the reason for a K8S executor death (as evident by the spark.kubernetes.ex

Re: Getting the ball started on a 2.4.6 release

I'd like to advocate for: https://issues.apache.org/jira/browse/SPARK-25515 and https://issues.apache.org/jira/browse/SPARK-29865 Two small QOL changes that make production use of Spark with Kubernetes much easier. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ -