Hi Jianshi, For YARN, there may be an issue with how a recently patch changes the accessibility of the shuffle files by the external shuffle service: https://issues.apache.org/jira/browse/SPARK-5655. It is likely that you will hit this with 1.2.1, actually. For this reason I would have to recommend that you use 1.2.2 when it is released, but for now you should use 1.2.0 for this specific use case.
-Andrew 2015-02-10 23:38 GMT-08:00 Reynold Xin <r...@databricks.com>: > I think we made the binary protocol compatible across all versions, so you > should be fine with using any one of them. 1.2.1 is probably the best since > it is the most recent stable release. > > On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <jianshi.hu...@gmail.com> > wrote: > >> Hi, >> >> I need to use branch-1.2 and sometimes master builds of Spark for my >> project. However the officially supported Spark version by our Hadoop admin >> is only 1.2.0. >> >> So, my question is which version/build of spark-yarn-shuffle.jar should I >> use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0) >> >> Thanks, >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > >
