Make sure ulimit is set system wide across the cluster (ulimit -a). Also reduce the number of partitions to a smaller number (say 200-500) to get ride of *the too many open files*.
Thanks Best Regards On Tue, Dec 16, 2014 at 3:54 PM, aecc <alessandroa...@gmail.com> wrote: > > Hi guys, > > It happens to me quite often that when the locality level of a task goes > further than LOCAL (NODE, RACK, etc), I get some of the following > exceptions: "too many files open", "encountered unregistered class id", > "cannot cast X to Y". > > I do not get any exceptions during shuffling (which means that kryo works > well). > > I'm running Spark 1.0.0 with the following characteristics: > > - 18 executors with 30G each > - Yarn client mode > - ulimit is defined in 500k > - Input data: hdfs file with 1000 partitions and 10 GB of size > > Please any hint would be appreciated > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Locality-level-and-Kryo-tp20708.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
