Yea I re-read the email again. It'd work in this case. The bigger problem is that it is much easier to maintain backward compatibility rather than dictating forward compatibility. For example, as Marcin said, if we come up with a slightly different shuffle layout to improve shuffle performance, we wouldn't be able to do that if we want to allow Spark 1.6 shuffle service to read something generated by Spark 2.1.
On Mon, Apr 18, 2016 at 1:59 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Mon, Apr 18, 2016 at 1:53 PM, Reynold Xin <r...@databricks.com> wrote: > > That's not the only one. For example, the hash shuffle manager has been > off > > by default since Spark 1.2, and we'd like to remove it in 2.0: > > https://github.com/apache/spark/pull/12423 > > If I understand things correctly, Mark's option B (running a single > shuffle service, the one from the older Spark release) would still > work, wouldn't it? > > You'd run into problems when Spark adds a new shuffle manager that is > not known to the old shuffle service, though. Perhaps at that time we > should investigate making the shuffle service more agnostic to the > app's shuffle manager. > > -- > Marcelo >
