[
https://issues.apache.org/jira/browse/SPARK-46860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938515#comment-17938515
]
Krzysztof Ruta commented on SPARK-46860:
----------------------------------------
Anyway I am not sure what's gonna happen with this PR, how long could it take
for somebody to review and (hopefully) integrate it.
If you wish to play with it or even use it in prod, you could build spark-core
yourself and then replace spark-core jar with yours. That's what I did. I can
provide you with some guidance if necessary.
> Credentials with https url not working for --jars, --files, --archives &
> --py-files options on spark-submit command
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-46860
> URL: https://issues.apache.org/jira/browse/SPARK-46860
> Project: Spark
> Issue Type: Task
> Components: k8s
> Affects Versions: 3.3.3, 3.5.0, 3.3.4
> Environment: Spark 3.3.3 deployed on K8s
> Reporter: Vikram Janarthanan
> Priority: Major
> Labels: pull-request-available
>
> We are trying to run the spark application by pointing the dependent files as
> well the main pyspark script from secure webserver
> We are looking for solution to pass the dependencies as well as pysaprk
> script from webserver.
> we have tried deploying the spark application from webserver to k8s cluster
> without username and password and it worked, but when tried with
> username/password we are facing "Exception in thread "{*}main"
> java.io.IOException: Server returned HTTP response code: 401 for URL:
> https://username:passw...@domain.com/application/pysparkjob.py{*}";
> *Working options on spark-submit:*
> spark-submit ......
> --repositories https://username:passw...@domain.com/repo1/repo
> --jars https://domain.com/jars/runtime.jar \
> --files https://domain.com/files/query.sql \
> --py-files [https://domain.com/pythonlib/pythonlib.zip] \
> https://domain.com/app1/pysparkapp.py
> Note: only repositories option works with username and password
> *Spark-submit using https url with username/password not working:*
> spark-submit ......
> --jars https://username:passw...@domain.com/jars/runtime.jar \
> --files https://username:passw...@domain.com/files/query.sql \
> --py-files
> https://username:passw...@domain.com[/pythonlib/pythonlib.zip|https://domain.com/pythonlib/pythonlib.zip]
> \
> https://username:passw...@domain.com/app1/pysparkapp.py
>
> Error :
> 25/01/23 09:19:57 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.io.IOException: Server returned HTTP response
> code: 401 for URL:
> https://username:passw...@domain.com/repository/spark-artifacts/pysparkdemo/1.0/pysparkdemo-1.0.tgz
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2000)
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
> at
> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:809)
> at
> org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264)
> at
> org.apache.spark.util.DependencyUtils$.$anonfun$downloadFileList$2(DependencyUtils.scala:233)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at
> scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
