I would have to check the Spark source code, but theoretically you can limit
the no of threads on the jvm level. Maybe spark does this.Alternatively, you
can use cgroups, but this introduces other complexity.
> On 10 Nov 2015, at 14:33, Peter Rudenko wrote:
>
> Hi i have a question: how does t
As i've tried cgroups - seems the isolation is done by percantage not by
cores number. E.g. i've set min share to 256 - i still see all 8 cores,
but i could only load only 20% of each core.
Thanks,
Peter Rudenko
On 2015-11-10 15:52, Saisai Shao wrote:
From my understanding, it depends on whethe
>From my understanding, it depends on whether you enabled CGroup isolation
or not in Yarn. By default it is not, which means you could allocate one
core but bump a lot of thread in your task to occupy the CPU resource, this
is just a logic limitation. For Yarn CPU isolation you may refer to this
po
Hi i have a question: how does the cores isolation works on spark on
yarn. E.g. i have a machine with 8 cores, but launched a worker with
--executor-cores 1, and after doing something like:
rdd.foreachPartition(=>{for all visible cores: burn core in a new tread})
Will it see 1 core or all 8 co
