I'm using Spark 1.0.1 on Yarn-Client mode. SortByKey always reports a FileNotFoundExceptions with messages says "too many open files".
I already set spark.shuffle.consolidateFiles to true:
conf.set("spark.shuffle.consolidateFiles", "true")
But it seems not working. What are the other possible reasons? How to fix
it?
Jianshi
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
