You might wanna try to assign more cores to the driver?! Sent from my iPhone
> On 20 Mar 2016, at 07:34, Jialin Liu <jaln...@lbl.gov> wrote: > > Hi, > I have set the partitions as 6000, and requested 100 nodes, with 32 > cores each node, > and the number of executors is 32 per node > > spark-submit --master $SPARKURL --executor-cores 32 --driver-memory > 20G --executor-memory 80G single-file-test.py > > > And I'm reading a 2.2 TB, the code, just has simple two steps, > rdd=sc.read > rdd.count > Then I checked the log file, and history server, it shows that the > count stage has a really large tasks launching range, e.g., > > 16/03/19 22:40:17 > 16/03/19 22:30:56 > > which is about 10 minutes, > Has anyone experienced this before? > Could you please let me know the reason and internal of Spark relating > to this issue, > and how to resolve it? Thanks much. > > Best, > Jialin > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
