Hi Team,
Thank you for clarifying about decommission ignore fetch failure behavior.
Previously I was using Executor Rolling and Decommision and Ignore
Decommission Fetch Failure as a solution for all the problems. I understand
that Executor rolling must be carefully tuned to minimize fetch failure
Hi, Arun.
Here are some answers to your questions.
First, the fetch failure is irrelevant to the Executor Rolling feature
because the plugin itself only asked the Spark scheduler to decommission
it, not terminate it. More specifically, it's independent from the
underlying Decommissioning feature'
Hi,
The crux of the matter here as I understand is " how should I be using
Executor Rolling, without triggering stage failures?"
The object of executor rolling is to replace decommissioning executors with
new ones while minimizing the impact on running tasks and stages. in k8s.
As mentioned
spa
unsubscribe
Hi Qian,
How in practice have you implemented image caching for the driver and
executor pods respectively?
Thanks
On Thu, 24 Aug 2023 at 02:44, Qian Sun wrote:
> Hi Mich
>
> I agree with your opinion that the startup time of the Spark on Kubernetes
> cluster needs to be improved.
>
> Regarding
Hi Team,
I am running Apache Spark 3.4.1 Application on K8s with the below
configuration related to executor rolling and Ignore Decommission Fetch
Failure.
spark.plugins: "org.apache.spark.scheduler.cluster.k8s.ExecutorRollPlugin"
spark.kubernetes.executor.rollInterval: "1800s"
spark.kubernetes.e
Hi, All.
Java 21 will be released in a month and Apache Spark master branch
(4.0.0-SNAPSHOT) achieved the first milestone (SPARK-43831: Build and Run
Spark on Java 21) Today.
1. JDK 21: https://openjdk.org/projects/jdk/21/
- 2023/08/24 Final Release Candidate
- 2023/09/19 General Availabilit
