Ron Gonzalez created SPARK-23527:
------------------------------------
Summary: Error with spark-submit and kerberos with TLS-enabled
Hadoop cluster
Key: SPARK-23527
URL: https://issues.apache.org/jira/browse/SPARK-23527
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.2.1
Environment: core-site.xml
<property>
<name>hadoop.security.key.provider.path</name>
<value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value>
</property>
hdfs-site.xml
<property>
<name>dfs.encryption.key.provider.uri</name>
<value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value>
</property>
Reporter: Ron Gonzalez
For current configuration of our enterprise cluster, I submit using
spark-submit:
./spark-submit --master yarn --deploy-mode cluster --class
org.apache.spark.examples.SparkPi --conf
spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar
../examples/jars/spark-examples_2.11-2.2.1.jar 10
I am getting the following problem:
18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
3351181 for svchdc236d on ha-hdfs:nameservice1
Exception in thread "main" java.lang.IllegalArgumentException:
java.net.UnknownHostException: host1.domain.com;host2.domain.com
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
at
org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52)
If I get rid of the other host for the properties so instead of
kms://ht...@host1.domain.com;host2.domain.com:16000/kms, I convert it to:
kms://ht...@host1.domain.com:16000/kms
it fails with a different error:
java.io.IOException: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
If I do the same thing using spark 1.6, it works so it seems like a
regression...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
