Hi,
The Linux and Windows builders have 32 GB of RAM, the Mac
builders 64 Gb.
We also run concurrent R CMD check's.
Here is a summary:
platform RAM nb of nb of concurrent
(Gb) cores R CMD check's
---------------------------------------------------
Linux (malbecs) 32 20 10
Windows (tokays) 32 40 24
Mac (meridas) 64 24 18
That's a lot of concurrency. And there is actually more
concurrency than that if you consider the fact that many
packages run things in parallel during 'R CMD check'.
Some packages are good citizens and limit the number of
cores to 1 or 2 only during 'R CMD check' but some packages
try to use all the cores that are available. This will have
a strong impact on the overall progress of the builds. We
don't have an easy way to identify those packages right now.
In average, based on our monitoring of the build machines
things seem to work ok i.e. the concurrent R CMD check's
don't seem to be competing too much to access resources.
But occasionally there could be too much competition. The
crazy big elapsed time compared to the relatively short user
and system times that you observed Kasper are likely to reflect
that. They could be the sign that the machine ran out of memory
and started swapping. Not because it happens to your package
means that your package uses too much memory. The swapping is
the result of the **cumulated** memory usage of all the
R CMD check's running at that moment. It could be worth checking
how much memory R CMD check'ing your package uses though.
The exact set of packages that are being R CMD check'ed at any
given time is in constant fluctuation and will also vary from
one day to the other. This would explain why some days you see
timeouts on some platforms and some days not. We don't have
an easy way to know which packages were competing with yours
during the 40 min window that 'R CMD check' was running on your
package until the build system declared a timeout. It's possible
(by looking at the BBS logs) but is time consuming.
We should probably add some memory at some point to the Windows
builders. 32 Gb is not enough to smoothly run 24 R CMD check's
concurrently.
H.
On 04/26/2018 08:48 AM, Diogo FT Veiga wrote:
Hi Daniel,
I have the same issue with my package (new contribution). I just finish
reviewing the package with the modifications requested.
I am having a warning because R CMD check is exceeding 5 min, but this is
happening only in the Windows machine.
In Linux and OSX the check finishes in <= 4min, while in Windows takes
~6min.
https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_spb-5Freports_maser-5Fbuildreport-5F20180425114748.html&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=1aMitB3PnVLoojx1lnj_UT_ZeKlJ_OcJDFT4D6BPXow&e=
Not sure how to proceed from here.
Thanks,
Diogo
On Thu, Apr 26, 2018 at 9:52 AM, Kasper Daniel Hansen <
kasperdanielhan...@gmail.com> wrote:
We have been working on the minfi package lately, with a move to a
DelayedArray backend.
Right now there are some weird issues regarding timings in R CMD check.
Leaving aside the issue that the tests (now disabled) and examples are too
slow, we get some very weird behaviour.
An example is the current (soon to be replace) build report of minfi 1.25.2
which prints
Examples with CPU or elapsed time > 5s
user system elapsed
preprocessFunnorm 99.388 0.632 148.554
combineArrays 64.104 2.120 68.329
bumphunter 62.540 1.392 64.107
preprocessNoob 43.944 0.016 44.955
preprocessQuantile 33.968 0.064 36.547
getAnnotation 31.072 0.024 31.126
compartments 18.668 0.188 18.871
minfiQC 10.124 6.628 1102.929
getSex 10.536 0.012 10.561
read.metharray 7.504 2.116 82.713
read.metharray.exp 9.076 0.032 10.592
mapToGenome-methods 4.648 0.548 163.648
mdsPlot 0.340 0.204 14.901
on Tokay (Linux). Note minfiQC which has an elapsed time which is crazy
high compared to user+system. Previous build report (which I didn't save)
had a timeout on all platforms with a semingly similar behaviour but with
the getSex function. The code did not change in the meantime. For today's
build we only see this on Linux, but yesterday all platforms were affected.
This is likely to be very hard to debug. But I am thinking memory issues:
this example requires loading an annotation package and a data package,
both of which are "big". How much RAM does the machines have and are
multiple R CMD check's run concurrently?
Best,
Kasper
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=R1DGN1kNpBZ4ZRBCTQzDPQlNYapuBNSYB4JTM6tO60w&e=
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=R1DGN1kNpBZ4ZRBCTQzDPQlNYapuBNSYB4JTM6tO60w&e=
--
Hervé Pagès
Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024
E-mail: hpa...@fredhutch.org
Phone: (206) 667-5791
Fax: (206) 667-1319
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
