On Tue, Apr 19, 2016 at 2:26 AM, Steve Loughran <ste...@hortonworks.com> wrote:
> > > On 18 Apr 2016, at 23:05, Marcelo Vanzin <van...@cloudera.com> wrote: > > > > On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <r...@databricks.com> > wrote: > >> The bigger problem is that it is much easier to maintain backward > >> compatibility rather than dictating forward compatibility. For example, > as > >> Marcin said, if we come up with a slightly different shuffle layout to > >> improve shuffle performance, we wouldn't be able to do that if we want > to > >> allow Spark 1.6 shuffle service to read something generated by Spark > 2.1. > > > > And I think that's really what Mark is proposing. Basically, "don't > > intentionally break backwards compatibility unless it's really > > required" (e.g. SPARK-12130). That would allow option B to work. > > > > If a new shuffle manager is created, then neither option A nor option > > B would really work. Moving all the shuffle-related classes to a > > different package, to support option A, would be really messy. At that > > point, you're better off maintaining the new shuffle service outside > > of YARN, which is rather messy too. > > > > > There's a WiP in YARN to move Aux NM services into their own CP, though > that doesn't address shared native libs, such as the leveldb support that > went into 1.6 > > > There's already been some fun with Jackson versions and that of Hadoop — > SPARK-12807; something that per-service classpaths would fix. > > would having separate CPs allow multiple spark shuffle JARs to be loaded, > as long as everything bonded to the right one? I just checked out https://issues.apache.org/jira/browse/YARN-1593. It's hard to say if it'd help or not, I wasn't able to find any design doc or patch attached to that JIRA. If there were a way to specify different JAR names/locations for starting the separate process, it would work but if the start happened by pointing to a full class name, that comes back to Option A, and we'd have to do a good chunk of name/version spacing in order to isolate.
