Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/969#issuecomment-46009163
Can you please be more specific on your description of this problem? Does
it only occur in yarn cluster mode (or is it both cluster and client mode)?
YarnClientSchedulerBackend I believe should handle loading that config in
client mode so I believe you are only seeing the problem in yarn-cluster mode.
I think the reason is that it was put in to replace the environment
variables for client mode (SPARK_YARN_DIST_ARCHIVES) and also when spark-submit
was added. Originally there was no config for specifying this in yarn-cluster
mode. I'm definitely fine with adding that support but I think your jira
should be changed to an improvement. I also had just filed SPARK-2139 but I
think that should just be combined with this and it documented here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
