I have not been able to run spark-shell in yarn-cluster mode since 1.5.0 due to the same issue described by [SPARK-9776]. Did this pull request fix the issue? https://github.com/apache/spark/pull/8947 I still have the same problem with 1.5.1 (I am running on HDP 2.2.6 with Hadoop 2.6) Thanks.
-Yao