A second advantage is that it allows individual Executors to go into GC pause (or even crash) and still allow other Executors to read shuffle data and make progress, which tends to improve stability of memory-intensive jobs.
On Thu, Jun 25, 2015 at 11:42 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > Hi Yash, > > One of the main advantages is that, if you turn dynamic allocation on, and > executors are discarded, your application is still able to get at the > shuffle data that they wrote out. > > -Sandy > > On Thu, Jun 25, 2015 at 11:08 PM, yash datta <sau...@gmail.com> wrote: > >> Hi devs, >> >> Can someone point out if there are any distinct advantages of using >> external shuffle service over yarn (runs on node manager as an auxiliary >> service >> >> https://issues.apache.org/jira/browse/SPARK-3797) instead of the >> default execution in the executor containers ? >> >> Please also mention if you have seen any differences having used both >> ways ? >> >> Thanks and Best Regards >> Yash >> >> -- >> When events unfold with calm and ease >> When the winds that blow are merely breeze >> Learn from nature, from birds and bees >> Live your life in love, and let joy not cease. >> > >
