So I added these settings in yarn-site.xml as well. Now I get a completely
different error, but atleast it seems like it is using the crypto library:
ExecutorLostFailure (executor 1 exited caused by one of the running tasks)
Reason: Unable to create executor due to Unable to register with external
shuffle server due to : java.lang.IllegalArgumentException: Authentication
failed.
at
org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
at
org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
at
org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
Any clue about this ?
On Wed, Jul 19, 2017 at 1:13 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> On Wed, Jul 19, 2017 at 1:10 PM, Udit Mehrotra
> <udit.mehrotr...@gmail.com> wrote:
> > Is there any additional configuration I need for external shuffle besides
> > setting the following:
> > spark.network.crypto.enabled true
> > spark.network.crypto.saslFallback false
> > spark.authenticate true
>
> Have you set these options on the shuffle service configuration too
> (which is the YARN xml config file, not spark-defaults.conf)?
>
> If you have there might be an issue, and you should probably file a
> bug and include your NM's log file.
>
> --
> Marcelo
>
