Hi Martin,
Thank you for the quick response. Now it works like a charm.
Best wishes,
Michael
On 27.05.2013 17:29, Martin Morgan wrote:
> On 05/26/2013 11:57 PM, Michael Stadler wrote:
>> Dear bioc core developers,
>>
>> We have recently (a few months back?) started to get error messages of
>> the type:
>>
>> libgomp: Thread creation failed: Resource temporarily unavailable
>>
>> when using ShortRead::FastqStreamer in combination with
>> parallel::parLapply (for example in QuasR::preprocessReads or
>> QuasR::qQCReport). These errors are not fully reproducible, are less
>> frequent when reducing the number of nodes in the cluster object and
>> disappear completely when FastqStreamer is used directly rather than
>> within a parLapply call of the sort:
>>
>> cl <- makeCluster(n)
>> results <- parLapply(cl, filenames, function(filename) {
>> fs <- FastqStreamer(filename, n=1e6)
>> while (length(chunks <- yield(fs)) != 0L) {
>> # do work
>> ...
>> }
>> close(fs)
>> })
>>
>> I guess that this could be a conflict of nested parallelization and/or a
>> resource limit of our setup. If I am not mistaken, FastqStreamer makes
>> use of OpenMP for parsing fastq files.
>>
>> My questions is how the "libgomp" error message can be avoided in the
>> above situations. Is it possible to switch off the use of OpenMP in
>> FastqStreamer and friends at run-time, other than switching off OpenMP
>> at compile time for R or the whole ShortRead package?
>
> Hi Michael --
>
> Your diagnosis sounds on target. It is possible to set the number of
> openMP threads, which I think will address the problem. You'd like to
> not interfere with others' use of openMP, so reset the thread count when
> your function exits. Along the lines of
>
> nthreads <- .Call(ShortRead:::.set_omp_threads, 1L)
> on.exit(.Call(ShortRead:::.set_omp_threads, nthreads))
>
> ShortRead's srapply function does this internally when it detects that
> the `parallel` package is in use (reasoning that mclapply and forked
> processes would be the common use case for single-machine parallel
> evaluation). I'll give some thought to more aggressively discovering
> when it is necessary to invoke FastqStreamer with this option set.
>
> Sorry for the trouble,
>
> Martin
>
>>
>> With kind regards,
>> Michael
>>
>>
>> My sessionInfo():
>> R version 3.0.1 (2013-05-16)
>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>
>> locale:
>> [1] C
>>
>> attached base packages:
>> [1] parallel stats graphics grDevices utils datasets methods
>> [8] base
>>
>> other attached packages:
>> [1] ShortRead_1.18.0 latticeExtra_0.6-24 Rsamtools_1.12.3
>> [4] lattice_0.20-15 Biostrings_2.28.0 GenomicRanges_1.12.3
>> [7] IRanges_1.18.1 BiocGenerics_0.7.0 RColorBrewer_1.0-5
>>
>> loaded via a namespace (and not attached):
>> [1] Biobase_2.20.0 bitops_1.0-5 grid_3.0.1 hwriter_1.3
>> stats4_3.0.1
>> [6] tcltk_3.0.1 tools_3.0.1 zlibbioc_1.6.0
>>
>> _______________________________________________
>> Bioc-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
>
--
--------------------------------------------
Michael Stadler, PhD
Head of Computational Biology
Friedrich Miescher Institute
Basel (Switzerland)
Phone : +41 61 697 6492
Fax : +41 61 697 3976
Mail : michael.stad...@fmi.ch
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
