Hello Spark Community, I am currently looking to optimize Apache Spark for ARM architecture by leveraging Scalable Vector Extensions (SVE). I am aware of Gluten and OAP which help optimize Spark performance externally, but the goal is to contribute directly to the Spark repository, and I’m seeking advice on the following aspects:
1. AVX Optimization Efforts: I am interested in understanding any existing optimization efforts within the Spark repository that focus on x86 architectures using Advanced Vector Extensions (AVX). 2. Target Components for ARM and SVE: I am looking for guidance on which components or areas within Spark might benefit the most from ARM architecture optimizations using SVE. Any recommendations on which parts of Spark would see the greatest performance improvements would be appreciated. 3. JNI/JNA for Offloading Compute Tasks: I am considering using JNI (Java Native Interface) or JNA (Java Native Access) to offload compute-intensive tasks to leverage SVE on ARM. Any insights on whether these approaches are suitable for integrating SVE optimizations with Spark, and if there are best practices or existing discussions on this topic would be helpful Additionally, if there are any existing discussions, proposals, or resources related to ARM optimizations within Spark, I would greatly appreciate pointers or references to these materials. Thank you for your guidance and support!
