seems apache-celeborn is also an option, if anyone has used this pls let me know.
thanks! On Mon, Mar 31, 2025 at 1:58 PM karan alang <karan.al...@gmail.com> wrote: > hello all - checking to see if anyone has any input on this > > thanks! > > > On Tue, Mar 25, 2025 at 12:22 PM karan alang <karan.al...@gmail.com> > wrote: > >> hello All, >> >> I have kubeflow Spark Operator installed on k8s and from what i >> understand - Spark Shuffle is not officially supported on kubernetes. >> >> Looking for feedback from the community on what approach is being taken >> to handle this issue - especially since dynamicAllocation cannot be >> enabled without Spark Shuffle. >> >> for eg. >> Does storing the shuffle data in PVC help ? >> >> Pls let me know. >> >> tia! >> >> >
