Matt, appreciate the update! On Wed, Dec 19, 2018 at 10:51 AM Matt Cheah <mch...@palantir.com> wrote:
> Hi everyone, > > > > Earlier this year, we proposed SPARK-25299 > <https://issues.apache.org/jira/browse/SPARK-25299>, proposing the idea > of using other storage systems for persisting shuffle files. Since that > time, we have been continuing to work on prototypes for this project. In > the interest of increasing transparency into our work, we have created a > progress > report document > <https://docs.google.com/document/d/1tglSkfblFhugcjFXZOxuKsCdxfrHBXfxgTs-sbbNB3c/edit?usp=sharing> > where you may find a summary of the work we have been doing, as well as > links to our prototypes on Github. We would ask that anyone who is very > familiar with the inner workings of Spark’s shuffle could provide feedback > and comments on our work thus far. We welcome any further discussion in > this space. You may comment in this e-mail thread or by commenting on the > progress report document. > > > > Looking forward to hearing from you. Thanks, > > > > -Matt Cheah > -- John
