Re: KubernetesLocalDiskShuffleDataIO mount path dependency doubt.

2023-08-11 Thread Arun Ravi
Hi Dongjoon, Thank you for sharing about Old Protocol and clearing my doubt. I was able to understand the difference between Spark 2 & 3. For now `KubernetesLocalDiskShuffleDataIO` works fine for me. Thanks, Arun Ravi M V B.Tech (Batch: 2010-2014) Computer Science and Engineering Govt. Model E

Re: KubernetesLocalDiskShuffleDataIO mount path dependency doubt.

2023-08-11 Thread Dongjoon Hyun
Hi, Arun. SPARK-35593 (Support shuffle data recovery on the reused PVCs) was Apache Spark 3.2.0 feature whose plugin follows only the legacy Spark shuffle directory structure to be safe. You can see the AS-IS test coverage in the corresponding `KubernetesLocalDiskShuffleDataIOSuite`. https://git

KubernetesLocalDiskShuffleDataIO mount path dependency doubt.

2023-08-11 Thread Arun Ravi
Hi Team, I am using the recently released shuffle recovery feature using `KubernetesLocalDiskShuffleDataIO` plugin class on Spark 3.4.1. Can someone explain why the mount path has spark-x/executor-x/ pattern dependency? I got this path detail from this PR