Hi Ted,
Thanks for reply.
I tried your patch but am having the same problem.
I ran:
./bin/pyspark --master yarn-client
>> sc.stop()
>> sc = SparkContext()
Same error dump as below.
Do I need to pass something to the new sparkcontext ?
Thanks,
Mike
From: Ted Yu <yuzhih...@gmail.com>
To: Michael V Le/Watson/IBM@IBMUS
Cc: user <user@spark.apache.org>
Date: 11/11/2015 01:55 PM
Subject: Re: Creating new Spark context when running in Secure YARN
fails
Looks like the delegation token should be renewed.
Mind trying the following ?
Thanks
diff --git
a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerB
index 20771f6..e3c4a5a 100644
---
a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
+++
b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
@@ -53,6 +53,12 @@ private[spark] class YarnClientSchedulerBackend(
logDebug("ClientArguments called with: " + argsArrayBuf.mkString(" "))
val args = new ClientArguments(argsArrayBuf.toArray, conf)
totalExpectedExecutors = args.numExecutors
+ // SPARK-8851: In yarn-client mode, the AM still does the credentials
refresh. The driver
+ // reads the credentials from HDFS, just like the executors and
updates its own credentials
+ // cache.
+ if (conf.contains("spark.yarn.credentials.file")) {
+ YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf)
+ }
client = new Client(args, conf)
appId = client.submitApplication()
@@ -63,12 +69,6 @@ private[spark] class YarnClientSchedulerBackend(
waitForApplication()
- // SPARK-8851: In yarn-client mode, the AM still does the credentials
refresh. The driver
- // reads the credentials from HDFS, just like the executors and
updates its own credentials
- // cache.
- if (conf.contains("spark.yarn.credentials.file")) {
- YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf)
- }
monitorThread = asyncMonitorApplication()
monitorThread.start()
}
On Wed, Nov 11, 2015 at 10:23 AM, mvle <m...@us.ibm.com> wrote:
Hi,
I've deployed a Secure YARN 2.7.1 cluster with HDFS encryption and am
trying
to run the pyspark shell using Spark 1.5.1
pyspark shell works and I can run a sample code to calculate PI just
fine.
However, when I try to stop the current context (e.g., sc.stop()) and
then
create a new context (sc = SparkContext()), I get the error below.
I have also seen errors such as: "token (HDFS_DELEGATION_TOKEN token 42
for
hadoop) can't be found in cache",
Does anyone know if it is possible to stop and create a new Spark context
within a single JVM process (driver) and have that work when dealing with
delegation tokens from Secure YARN/HDFS?
Thanks.
15/11/11 10:19:53 INFO yarn.Client: Setting up container launch context
for
our AM
15/11/11 10:19:53 INFO yarn.Client: Setting up the launch environment for
our AM container
15/11/11 10:19:53 INFO yarn.Client: Credentials file set to:
credentials-37915c3e-1e90-44b9-add1-521598cea846
15/11/11 10:19:53 INFO yarn.YarnSparkHadoopUtil: getting token for
namenode:
hdfs://test6-allwkrbsec-001:9000/user/hadoop/.sparkStaging/application_1446695132208_0042
15/11/11 10:19:53 ERROR spark.SparkContext: Error initializing
SparkContext.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation
Token
can be issued only with kerberos or web authentication
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken
(FSNamesystem.java:6638)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken
(NameNodeRpcServer.java:563)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken
(ClientNamenodeProtocolServerSideTranslatorPB.java:987)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos
$ClientNamenodeProtocol$2.callBlockingMethod
(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call
(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke
(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken
(ClientNamenodeProtocolTranslatorPB.java:933)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod
(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke
(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1044)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken
(DistributedFileSystem.java:1543)
at
org.apache.hadoop.fs.FileSystem.collectDelegationTokens
(FileSystem.java:530)
at
org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens
(DistributedFileSystem.java:2228)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun
$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun
$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes
(YarnSparkHadoopUtil.scala:123)
at
org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval
(Client.scala:495)
at
org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:528)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext
(Client.scala:628)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start
(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start
(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
at
org.apache.spark.api.java.JavaSparkContext.<init>
(JavaSparkContext.scala:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance
(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance
(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance
(Constructor.java:526)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
at
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:214)
at
py4j.commands.ConstructorCommand.invokeConstructor
(ConstructorCommand.java:79)
at
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Creating-new-Spark-context-when-running-in-Secure-YARN-fails-tp25361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
