Hi guys, It happens to me quite often that when the locality level of a task goes further than LOCAL (NODE, RACK, etc), I get some of the following exceptions: "too many files open", "encountered unregistered class id", "cannot cast X to Y".
I do not get any exceptions during shuffling (which means that kryo works well). I'm running Spark 1.0.0 with the following characteristics: - 18 executors with 30G each - Yarn client mode - ulimit is defined in 500k - Input data: hdfs file with 1000 partitions and 10 GB of size Please any hint would be appreciated -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Locality-level-and-Kryo-tp20708.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
