Hello Team, I was working with Spark 3.2 and Hadoop 2.7.6 and writing to MinIO object storage . It was slower when compared to writing to MapR FS with the above tech stack. Then moved on to a later upgraded version of Spark 3.5.2 and Hadoop 4.3.1 which started writing to MinIO with V2 fileoutputcommitter and checked the performance which is worse than old tech stack. Then tried using magic committer and it came out slower than V2 so with the latest tech stack the performance is downgraded. With Spark 3.2 and Hadoop 2.7.6 I was using Spark StandAlone cluster but with Spark 3.5.2 and Hadoop 3.4.1 used Kuberentes as cluster master.
Could some please assist .
