Hey guys, My colleague tried to post a question twice but somehow it doesn't show up in or emails, but it does exist in the archive<https://lists.apache.org/thread/9g659kpwc11y50kwqjc7pbvgyjfcp90b>. So, I will post the question here again.
We are running into some issues while attempting graceful decommissioning of executors. We are running spark-thriftserver (3.2.0) on Kubernetes (GKE 1.20.15-gke.2500). We enabled: - spark.decommission.enabled - spark.storage.decommission.rddBlocks.enabled - spark.storage.decommission.shuffleBlocks.enabled - spark.storage.decommission.enabled and set spark.storage.decommission.fallbackStorage.path to a path in our bucket. The logs from the driver seems to suggest the decommissioning process started but then unexpectedly exited and failed while the executor logs seem to suggest that decommissioning was successful. Attached are the error logs: https://gist.github.com/yeachan153/9bfb2f0ab9ac7f292fb626186b014bbf Thanks in advance.
