Hi, please see Sean's answer and please read about parallelism in spark. Regards, Gourav Sengupta
On Mon, Jul 11, 2022 at 10:12 AM Tufan Rakshit <tufan...@gmail.com> wrote: > so as an average every 4 core , you get back 3.6 core in Yarn , but you > can use only 3 . > in Kubernetes you get back 3.6 and also can use 3.6 > > Best > Tufan > > On Mon, 11 Jul 2022 at 11:02, Yong Walt <yongw...@gmail.com> wrote: > >> We were using Yarn. thanks. >> >> On Sun, Jul 10, 2022 at 9:02 PM Tufan Rakshit <tufan...@gmail.com> wrote: >> >>> Mainly depends what your cluster manager Yarn or kubernates ? >>> Best >>> Tufan >>> >>> On Sun, 10 Jul 2022 at 14:38, Sean Owen <sro...@gmail.com> wrote: >>> >>>> Jobs consist of tasks, each of which consumes a core (can be set to >1 >>>> too, but that's a different story). If there are more tasks ready to >>>> execute than available cores, some tasks simply wait. >>>> >>>> On Sun, Jul 10, 2022 at 3:31 AM Yong Walt <yongw...@gmail.com> wrote: >>>> >>>>> given my spark cluster has 128 cores totally. >>>>> If the jobs (each job was assigned only one core) I submitted to the >>>>> cluster are over 128, what will happen? >>>>> >>>>> Thank you. >>>>> >>>>
