> On the other hand, [...], you can spawn 20 workers on each of the 10 hosts:
>
> makePSOCKcluster(names = rep(c('Host01', ..., 'Host10'), each = 20))Unfortunately, this will most likely not work because it will require 200 open connections - one for each worker - but R limits you to 125 (see ?base::connections): > A maximum of 128 connections can be allocated (not necessarily open) at any > one time. Three of these are pre-allocated (see stdout). The OS will impose > limits on the numbers of connections of various types, but these are usually > larger than 125. Depending on the system, you might be able to increase this by rebuilding R from source after editing a hard-coded constant. I've verified that this worked on a local Ubuntu 16.04 system. See https://github.com/HenrikBengtsson/Wishlist-for-R/issues/28 for details about this problem. /Henrik On Fri, Jun 26, 2020 at 12:11 AM Ivan Krylov <krylov.r...@gmail.com> wrote: > > On Thu, 25 Jun 2020 00:29:42 +0000 > "Silva, Eder David Borges da" <eder.si...@corteva.com> wrote: > > > I have the HPC, with 10 nodes, and each node with 20 cores in UNIX OS. > > > cl <- makePSOCKcluster(names=c('Host01', ... , 'Host10) > > > This code is the best way for use all machine power? > > The code as written will create one worker _process_ on each of the > hosts. What happens next depends on the code to be running and the way > R is installed. > > The code may or may not be written to take advantage of multi-core CPUs > (e.g. using OpenMP). In particular, if R is linked with a > multi-threaded BLAS (such as OpenBLAS or MKL) and uses matrix algebra > during the computation, it may spawn multiple _threads_ to utilise the > CPU better. Whether it succeeds depends on multiple factors, including > the size of the task. On occasion I noticed OpenBLAS threads spending > most of their time in sched_yield() system call, making the kernel do a > lot of unnecessary work, and set the environment variable > OPENBLAS_NUM_THREADS=1 to use only one thread instead. > > On the other hand, if the computation is purely single-threaded (or you > disabled the multi-threaded behaviour of OpenMP or BLAS for some > reason), you can spawn 20 workers on each of the 10 hosts: > > makePSOCKcluster(names = rep(c('Host01', ..., 'Host10'), each = 20)) > > You can also try to combine the two approaches by limiting the number > of working threads to a sensible value which results in the threads > spending most of the time computing things (instead of waiting for more > work busy-looping on sched_yield()), then spawning as many processes as > required to utilise all of the cores. > > -- > Best regards, > Ivan > > ______________________________________________ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
