Great - using `local` solves my problem here.
> You said 'for Windows here serial' but a SerialParam() just runs lapply and
> would not have problems finding f
Ah, I assumed that the registry under Windows would also have a MulticoreParam
as default which would then dispatch to a SerialParam -- but I now see that it
actually has a SnowParam as default. Even better!
Thanks a lot,
Ludwig
--
Dr. Ludwig Geistlinger
CUNY School of Public Health
________________________________________
From: Martin Morgan <martin.mor...@roswellpark.org>
Sent: Wednesday, January 31, 2018 6:51 PM
To: Ludwig Geistlinger; bioc-devel@r-project.org
Subject: Re: [Bioc-devel] BiocParallel: windows vs. mac/linux behavior
On 01/31/2018 06:39 PM, Ludwig Geistlinger wrote:
> Hi,
>
>
> I am currently considering the following snippet:
>
>
>> data.ids <- paste0("d", 1:5)
>> f <- function(x) paste("dataset", x, sep=" = ")
>> res <- BiocParallel::bplapply(data.ids, function(d) f(d))
>
>
> Using a recent R-devel on both a Linux machine and a Mac machine, this works
> fine.
>
>
> However, on a Windows R-devel this throws:
>
>
> Error: BiocParallel errors
>
> element index: 1, 2, 3, 4, 5 ....
>
> first error: could not find function "f"
>
>
> Is this a bug or is this related to the different ways in which parallel (for
> windows here serial) computation is carried out?
>
Windows runs brand-new processes, whereas linux / mac use forked
processes that share the original process's memory. So Windows doesn't
know about the variables defined in the global environment. You can
mimick this on non Windows using SnowParam(), e.g.,
register(bpstart(SnowParam(2)) ## set up a cluster for the duration...
> res <- BiocParallel::bplapply(data.ids, function(d) f(d))
Error: BiocParallel errors
element index: 1, 2, 3, 4, 5
first error: could not find function "f"
The rule is that the environment of the FUN gets serialized to the
worker, up to the global environment. So
local({
f <- function(x) paste("dataset", x, sep=" = ")
BiocParallel::bplapply(data.ids, function(d) f(d))
})
works (FUN is defined in the local environment, the local environment
contains f and is serialized to the worker). This also implies that
BiocParallel will find functions that are in the same package as FUN.
A tweak
BiocParallel::bplapply(data.ids, f)
also works, because f (but not the .GlobalEnv in which it was defined)
is serialized.
You said 'for Windows here serial' but a SerialParam() just runs lapply
and would not have problems finding f.
Martin
>
> Thanks,
>
> Ludwig
>
>
>
> --
> Dr. Ludwig Geistlinger
> CUNY School of Public Health
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
This email message may contain legally privileged and/or confidential
information. If you are not the intended recipient(s), or the employee or
agent responsible for the delivery of this message to the intended
recipient(s), you are hereby notified that any disclosure, copying,
distribution, or use of this email message is prohibited. If you have received
this message in error, please notify the sender immediately by e-mail and
delete this email message from your computer. Thank you.
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
