Hi all, May I ask a question about executor memory setting? I was running PageRank with input size 2.8GB on one workstation for testing. I gave PageRank one executor.
In case 1, I set --executor-cores to 4, and --executor-memory to 1GB, the stage (stage 2) completion time is 14 min, the the detailed stage info is below: In case 2, I set --executor-cores to 4, and --executor-memory to 6GB, the stage (stage 2) completion time is 34 min, the the detailed stage info is below: I am totally confused why when executor-memory gets larger, the stage completion time is more than two times slower? From the web UI, I found that when executor memory is 6GB, the shuffle spill (Disk) per task is smaller, which means fewer IO operations, but weirdly, the task completion time is longer though. Could anyone give me some hints? Great thanks!
