Re: Design for continuous processing shuffle

2018-05-07 Thread Yuanjian Li
Hi Joseph and devs, Happy to see the discussion of CP shuffle, as comment in https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16245556

Design for continuous processing shuffle

2018-05-04 Thread Joseph Torres
Hi all, A few of us have been working on a design for how to do shuffling in continuous processing. Feel free to chip in if you have any comments or questions. doc: https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE continuous processing SPIP: https://issues.apache.o