Dear all, We reopened the existing JIRA entry https://issues.apache.org/jira/browse/SPARK-3785 to support off-loading computations to a GPU by adding a description for our prototype. We are working to effectively and easily exploit GPUs on Spark at http://github.com/kiszk/spark-gpu. Please also visit our project page http://kiszk.github.io/spark-gpu/.
For now, we added a new format for a partition in an RDD, which is a column-based structure in an array format, in addition to the current Iterator[T] format with Seq[T]. This reduces data serialization/deserialization and copy overhead between CPU and GPU. Our prototype achieved more than 3x performance improvement for a simple logistic regression program using a NVIDIA K40 card. This JIRA entry (SPARK-3785) includes a link to a design document. We are very glad to hear valuable feedback/suggestions/comments and to have great discussions to exploit GPUs in Spark. Best Regards, Kazuaki Ishizaki
