[
https://issues.apache.org/jira/browse/SPARK-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Parente closed SPARK-19105.
---------------------------------
Resolution: Invalid
> yarn/Client.scala copyToRemote does not include keytab destination name
> -----------------------------------------------------------------------
>
> Key: SPARK-19105
> URL: https://issues.apache.org/jira/browse/SPARK-19105
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.1.0
> Environment: YARN in client mode
> Reporter: Peter Parente
>
> When I specify --principal user@REALM and --keytab /some/path/user.keytab, I
> see the following in my app staging directory on HDFS:
> {code}
> -rw-r--r-- 3 user supergroup 68 2017-01-06 03:59 user.keytab
> -rw-r--r-- 3 user supergroup 73502 2017-01-06 03:59 __spark_conf__.zip
> -rw-r--r-- 3 user supergroup 189767340 2017-01-06 03:59
> __spark_libs__4440821503780683972.zip
> -rw-r--r-- 3 user supergroup 91275 2017-01-06 03:59 py4j-0.10.3-src.zip
> -rw-r--r-- 3 user supergroup 440385 2017-01-06 03:59 pyspark.zip
> {code}
> I also see that my spark.yarn.keytab config value has changed to
> user.keytab-54ee5192-43d0-41b5-ba50-1181ece26961 by the yarn client to ensure
> the keytab is unique within the app staging directory. However, from the
> directory listing above, it's clear that the file written does not match this
> new name. As a result, when it comes time to renew the Kerberos ticket,
> AMDelegationTokenRenewer fails to find the keytab under the UUID-suffixed
> name and also fails to renew the tickets.
> The problem looks to be in one call to [copyFileToRemote in
> yarn/Client.java|https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L482]
> that leaves off the destination filename param. The other calls in that
> object which use copyFileToRemote and have a custom destination name all
> provide this parameter (e.g.,
> https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L652).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
