So i am using the standard documentation here for configuring external shuffle service:
https://spark.apache.org/docs/latest/running-on-yarn.html# configuring-the-external-shuffle-service I can see on the cluster that following jars are present as well: /usr/lib/spark/yarn/spark-2.2.0-yarn-shuffle.jar Is there any additional configuration I need for external shuffle besides setting the following: spark.network.crypto.enabled true spark.network.crypto.saslFallback false spark.authenticate true On Wed, Jul 19, 2017 at 12:51 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Well, how did you install the Spark shuffle service on YARN? It's not > part of YARN. > > If you really have the Spark 2.2 shuffle service jar deployed in your > YARN service, then perhaps you didn't configure it correctly to use > the new auth mechanism. > > On Wed, Jul 19, 2017 at 12:47 PM, Udit Mehrotra > <udit.mehrotr...@gmail.com> wrote: > > Sorry about that. Will keep the list in my replies. > > > > So, just to clarify I am not using an older version of sparks shuffle > > service. This is a brand new cluster with just Spark 2.2.0 installed > > alongside hadoop 2.7.3. Could there be anything else I am missing, or I > can > > try differently ? > > > > > > Thanks ! > > > > > > On Wed, Jul 19, 2017 at 12:03 PM, Marcelo Vanzin <van...@cloudera.com> > > wrote: > >> > >> Please include the list on your replies, so others can benefit from > >> the discussion too. > >> > >> On Wed, Jul 19, 2017 at 11:43 AM, Udit Mehrotra > >> <udit.mehrotr...@gmail.com> wrote: > >> > Hi Marcelo, > >> > > >> > Thanks a lot for confirming that. Can you explain what you mean by > >> > upgrading > >> > the version of shuffle service ? Wont it automatically use the > >> > corresponding > >> > class from spark 2.2.0 to start the external shuffle service ? > >> > >> That depends on how you deploy your shuffle service. Normally YARN > >> will have no idea that your application is using a new Spark - it will > >> still have the old version of the service jar in its classpath. > >> > >> > >> -- > >> Marcelo > > > > > > > > -- > Marcelo >
