Well, how did you install the Spark shuffle service on YARN? It's not part of YARN.
If you really have the Spark 2.2 shuffle service jar deployed in your YARN service, then perhaps you didn't configure it correctly to use the new auth mechanism. On Wed, Jul 19, 2017 at 12:47 PM, Udit Mehrotra <udit.mehrotr...@gmail.com> wrote: > Sorry about that. Will keep the list in my replies. > > So, just to clarify I am not using an older version of sparks shuffle > service. This is a brand new cluster with just Spark 2.2.0 installed > alongside hadoop 2.7.3. Could there be anything else I am missing, or I can > try differently ? > > > Thanks ! > > > On Wed, Jul 19, 2017 at 12:03 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: >> >> Please include the list on your replies, so others can benefit from >> the discussion too. >> >> On Wed, Jul 19, 2017 at 11:43 AM, Udit Mehrotra >> <udit.mehrotr...@gmail.com> wrote: >> > Hi Marcelo, >> > >> > Thanks a lot for confirming that. Can you explain what you mean by >> > upgrading >> > the version of shuffle service ? Wont it automatically use the >> > corresponding >> > class from spark 2.2.0 to start the external shuffle service ? >> >> That depends on how you deploy your shuffle service. Normally YARN >> will have no idea that your application is using a new Spark - it will >> still have the old version of the service jar in its classpath. >> >> >> -- >> Marcelo > > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
