Hmm... I tried this with the new shuffle service (I generally have an old one running) and also see failures. I also noticed some odd things in your logs that I'm also seeing in mine, but it's better to track these in a bug instead of e-mail.
Please file a bug and attach your logs there, I'll take a look at this. On Thu, Jul 20, 2017 at 2:06 PM, Udit Mehrotra <udit.mehrotr...@gmail.com> wrote: > Hi Marcelo, > > I ran with setting DEBUG level logging for 'org.apache.spark.network.crypto' > for both Spark and Yarn. > > However, the DEBUG logs still do not convey anything meaningful. Please find > it attached. Can you please take a quick look, and let me know if you see > anything suspicious ? > > If not, do you think I should open a JIRA for this ? > > Thanks ! > > On Wed, Jul 19, 2017 at 3:14 PM, Marcelo Vanzin <van...@cloudera.com> wrote: >> >> Hmm... that's not enough info and logs are intentionally kept silent >> to avoid flooding, but if you enable DEBUG level logging for >> org.apache.spark.network.crypto in both YARN and the Spark app, that >> might provide more info. >> >> On Wed, Jul 19, 2017 at 2:58 PM, Udit Mehrotra >> <udit.mehrotr...@gmail.com> wrote: >> > So I added these settings in yarn-site.xml as well. Now I get a >> > completely >> > different error, but atleast it seems like it is using the crypto >> > library: >> > >> > ExecutorLostFailure (executor 1 exited caused by one of the running >> > tasks) >> > Reason: Unable to create executor due to Unable to register with >> > external >> > shuffle server due to : java.lang.IllegalArgumentException: >> > Authentication >> > failed. >> > at >> > >> > org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125) >> > at >> > >> > org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157) >> > at >> > >> > org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105) >> > at >> > >> > org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118) >> > >> > Any clue about this ? >> > >> > >> > On Wed, Jul 19, 2017 at 1:13 PM, Marcelo Vanzin <van...@cloudera.com> >> > wrote: >> >> >> >> On Wed, Jul 19, 2017 at 1:10 PM, Udit Mehrotra >> >> <udit.mehrotr...@gmail.com> wrote: >> >> > Is there any additional configuration I need for external shuffle >> >> > besides >> >> > setting the following: >> >> > spark.network.crypto.enabled true >> >> > spark.network.crypto.saslFallback false >> >> > spark.authenticate true >> >> >> >> Have you set these options on the shuffle service configuration too >> >> (which is the YARN xml config file, not spark-defaults.conf)? >> >> >> >> If you have there might be an issue, and you should probably file a >> >> bug and include your NM's log file. >> >> >> >> -- >> >> Marcelo >> > >> > >> >> >> >> -- >> Marcelo > > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
