yes apache celeborn may be useful. You need to do some research though. https://celeborn.apache.org/
Have a look at this link as well Spark Executor Shuffle Storage Options <https://iomete.com/resources/k8s/spark-executor-shuffle-storage-options>HTHDr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> On Mon, 31 Mar 2025 at 22:12, karan alang <karan.al...@gmail.com> wrote: > seems apache-celeborn is also an option, if anyone has used this pls let > me know. > > thanks! > > > On Mon, Mar 31, 2025 at 1:58 PM karan alang <karan.al...@gmail.com> wrote: > >> hello all - checking to see if anyone has any input on this >> >> thanks! >> >> >> On Tue, Mar 25, 2025 at 12:22 PM karan alang <karan.al...@gmail.com> >> wrote: >> >>> hello All, >>> >>> I have kubeflow Spark Operator installed on k8s and from what i >>> understand - Spark Shuffle is not officially supported on kubernetes. >>> >>> Looking for feedback from the community on what approach is being taken >>> to handle this issue - especially since dynamicAllocation cannot be >>> enabled without Spark Shuffle. >>> >>> for eg. >>> Does storing the shuffle data in PVC help ? >>> >>> Pls let me know. >>> >>> tia! >>> >>> >>
