We raised this SPIP ticket in https://issues.apache.org/jira/browse/SPARK-30602 earlier this year. Since then, we have progressed in multiple fronts, including:
* Our work is published in VLDB 2020. The final version of the paper is attached in the SPIP ticket. * We have further enhanced and productionized this work at LinkedIn, and have enabled production flows adopting the new push-based shuffle mechanism, with good results. * We have recently also ported our push-based shuffle changes to OSS Spark master branch, so other people can potentially try it out. Details of this branch is in this doc <https://docs.google.com/document/d/16yOfI8P_O3V6hx_FnWT22jeDIItgXuXfaDAV0fJDTqQ/edit#> * The SPIP doc <https://docs.google.com/document/d/1mYzKVZllA5Flw8AtoX7JUcXBOnNIDADWRbJ7GI6Y71Q/edit> is also further updated reflecting more recent designs. * We have also discussed with multiple companies who share similar interest in this work. We would like to resume the discussion of this SPIP in the community, and push for a voting on this. ----- Min Shen Staff Software Engineer LinkedIn -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
