Dear Community,
Im running into an error in the vignette of the “
*SingleCellAlleleExperiment*” Software package on the *palomino7/8* windows
distribution.
Error report in short:
Quitting from lines 229-238 [unnamed-chunk-8] (scae_intro.Rmd)
Error: processing vignette 'scae_intro.Rmd' failed w
Hello Anna,
The speed of parallel computing depends on many factors. To avoid any
potential confounders, Please try to use this code for timing (assuming you
still have all the variables you used in your example)
```
parallel_param <- SnowParam(workers = ncores, type = "SOCK", tasks =
My motivation for using distributed memory was that my package is also
accessible on Windows. Is it better to use shared memory as default but
check the user's system and then switch to socket only if necessary?
Regarding the real data. I have 68 samples (rows) of methylation EPIC array
data (850K
Dear Anna,
According to the documentation of "BiocParallelParam", SnowParam() is a
subclass suitable for distributed memory (e.g. cluster) computing. If you're
running your code on a simpler machine with shared memory (e.g. your PC),
you're probably better off using MulticoreParam() instead. He
Hi all!
I'm switching from the base R *parallel* package to *BiocParallel* for my
Bioconductor submission and I have two questions. First, I wanted advice on
whether I've implemented load balancing correctly. Second, I've noticed
that the running time is about 15% longer with BiocParallel. Any ide
’ll try with
> this one,
>
> Thanks again to everyone and have a nice day
>
>
>
> Giulia
>
>
>
> From: Martin Morgan
> Date: Thursday, July 7, 2022 at 14:28
> To: Giulia Pais , Henrik Bengtsson
>
> Cc: bioc-devel@r-project.org
> Subject: Re: [Bioc
, sorry.
Thank you
From: Vincent Carey
Date: Thursday, July 7, 2022 at 11:40
To: Giulia Pais
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] BiocParallel and Shiny
Interesting question. Have you looked at
https://shiny.rstudio.com/articles/progress.html ...? There is
also a file called
Hello,
I have a question on the use of BiocParallel with Shiny: I would like to show a
progress bar on the UI much like the standard progress bar that can be set in
functions like bplapply() � is it possible to do it and how? I haven�t found
anything on the topic in the documentation unfortunate
Good day,
I am not sure how to fix my package properly, even with the good example. A
link to the specific part of my function is
https://github.com/DarioS/ClassifyR/blob/e35899caceb401691990136387a517f4c3b57d5e/R/runTests.R#L567
and the example in the help page of runTestsEasyHard function tri
The question is a bit abstract for me to understand and it might be better to
point to actual code in a git repository or similar...
Inside a package, something like
fun = function(x, y, ...) {
c(x, y, length(as.list(...)))
}
user_visible <- function(x, ...) {
y = 1
internal_functi
Good day,
Thanks for the examples which demonstrate the issue. Do you have other
recommendations if, inside the loop, another function in the package is being
called and the variable being passed is the ellipsis? There are only a couple
of variables which might be provided by the user collected
Windows uses separate processes that do not share memory (SnowParam()), whereas
linux / mac by default use forked processes that share the original memory
(MulticoreParam()). So
> y = 1
> param = MulticoreParam()
> res = bplapply(1:2, function(x) y, BPPARAM=param)
works because the function can
Good day,
I have a loop in a function of my R package which by default uses bpparam() to
set the framework used for parallelisation. On Windows, I see the error
Error: BiocParallel errors
element index: 1, 2, 3, 4, 5, 6, ...
first error: object 'selParams' not found
This error does not happ
yes it would be useful to post this to R-devel as a 'using
parallel::makeCluster() question, removing BiocParallel from the
equation, where some general insight might be had...
Martin
On 06/13/2018 05:00 PM, Dario Strbenac wrote:
Good day,
I couldn't get a working param object. It never comp
Good day,
I couldn't get a working param object. It never completes the command
param = bpstart(SnowParam(2, manager.hostname = "144.130.152.1", manager.port =
2559))
I obtained the IP address by typing "My IP address" into Google and it gave me
the address shown. I used netstat -an and
P
It's more likely that it never starts, probably because it tries to
create socket connections on ports that are not available, or perhaps
because the file path to the installed location of the BiocParallel
package is on a network share, or the 'master' node needs to be
specified with an IP addr
Good day,
I was interested how the performance of my package is on a 32-bit Windows
computer because I'm going to give a workshop about it soon and some people
might bring old laptops. I found that using SnowParam with workers set to more
than 1 never finishes. The minimal code to cause the iss
er; bioc-devel@r-project.org
Subject: Re: [Bioc-devel] BiocParallel: windows vs. mac/linux behavior
On 01/31/2018 06:39 PM, Ludwig Geistlinger wrote:
> Hi,
>
>
> I am currently considering the following snippet:
>
>
>> data.ids <- paste0("d", 1:5)
>
On 01/31/2018 06:39 PM, Ludwig Geistlinger wrote:
Hi,
I am currently considering the following snippet:
data.ids <- paste0("d", 1:5)
f <- function(x) paste("dataset", x, sep=" = ")
res <- BiocParallel::bplapply(data.ids, function(d) f(d))
Using a recent R-devel on both a Linux machine a
Hi,
I am currently considering the following snippet:
> data.ids <- paste0("d", 1:5)
> f <- function(x) paste("dataset", x, sep=" = ")
> res <- BiocParallel::bplapply(data.ids, function(d) f(d))
Using a recent R-devel on both a Linux machine and a Mac machine, this works
fine.
However, on
0 PM
To: Ludwig Geistlinger; Gabe Becker; Vincent Carey
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] BiocParallel and AnnotationDbi: database disk image
is malformed
On 01/19/2018 02:24 PM, Ludwig Geistlinger wrote:
> I apologize if I haven't been specific enough - however, I am al
Ludwig Geistlinger
CUNY School of Public Health
From: Bioc-devel on behalf of Martin Morgan
Sent: Friday, January 19, 2018 1:54 PM
To: Gabe Becker; Vincent Carey
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] BiocParallel and AnnotationDbi: databas
1] compiler_3.4.1pillar_1.1.0 Biostrings_2.46.0 XML_3.98-1.9
[25] pkgconfig_2.0.1
--
Dr. Ludwig Geistlinger
CUNY School of Public Health
From: Bioc-devel on behalf of Martin Morgan
Sent: Friday, January 19, 2018 1:54 PM
To: Gabe Becker
On 01/19/2018 12:37 PM, Gabe Becker wrote:
IT seems like you could also force a copy of the reference object via
$copy() and then force a refresh of the conn slot by assigning a
new db connection into it.
I'm having trouble confirming that this would work, however, because I
actually can't repro
On 01/19/2018 12:23 PM, Vincent Carey wrote:
good question
some of the discussion on
http://sqlite.1065341.n5.nabble.com/Parallel-access-to-read-only-in-memory-database-td91814.html
seems relevant.
converting the relatively small annotation package content to pure R
read-only tables on the ma
IT seems like you could also force a copy of the reference object via
$copy() and then force a refresh of the conn slot by assigning a
new db connection into it.
I'm having trouble confirming that this would work, however, because I
actually can't reproduce the error. The naive way works for me on
good question
some of the discussion on
http://sqlite.1065341.n5.nabble.com/Parallel-access-to-read-only-in-memory-database-td91814.html
seems relevant.
converting the relatively small annotation package content to pure R
read-only tables on the master before parallelizing
might be very simple?
Hi,
Within a package I am developing, I would like to enable parallel probe to gene
mapping for a compendium of microarray datasets.
This accordingly makes use of annotation packages such as hgu133a.db, which in
turn connect to the SQLite database via AnnotationDbi.
When running in multi-core
On 12/30/2017 04:08 PM, Ludwig Geistlinger wrote:
Hi,
I'm currently playing around with progress bars in BiocParallel - which is a
great package! ;-)
For demonstration, I'm using the example code from DESeq2::DESeq.
library(DESeq2)
library(BiocParallel)
f <- function(mu)
{
cnts <- ma
Hi,
I'm currently playing around with progress bars in BiocParallel - which is a
great package! ;-)
For demonstration, I'm using the example code from DESeq2::DESeq.
library(DESeq2)
library(BiocParallel)
f <- function(mu)
{
cnts <- matrix(rnbinom(n=1000, mu=mu, size=1/0.5), ncol=10)
it looks like manager.hostname="127.0.0.1" does the trick.
> options(bphost="127.0.0.1")
> library(BiocParallel)
> register(MulticoreParam(2))
also works. Many thanks.
On Wed, Nov 22, 2017 at 1:25 PM, Martin Morgan <
martin.mor...@roswellpark.org> wrote:
> Does it help to use the argument ma
Does it help to use the argument manager.hostname="localhost" (or indeed
the name of your computer) following
https://support.bioconductor.org/p/88307/#88874
? and can you clarify about how you explored using different ports? See
also ?SnowParam section 'Global Options'
Martin
On 11/22/2017
from example(bplapply), after register(MulticoreParam(2, timeout=5))
bplppl> bplapply(1:10, fun)
*Error in socketConnection(host, port, TRUE, TRUE, "a+b", timeout =
timeout) : *
* cannot open the connection*
*In addition: Warning message:*
*In socketConnection(host, port, TRUE, TRUE, "a+b", t
>
> On 09/05/2015 12:10 AM, Obenchain, Valerie wrote:
>> Hi Robert,
>>
>> Thanks for reporting the bug. The problem was with how 'X' was split
>> before dispatching to bplapply() and affected both SerialParam and
>> SnowParam. Now fixed in release (1.2.21) and devel (1.3.52).
0 AM, Obenchain, Valerie wrote:
Hi Robert,
Thanks for reporting the bug. The problem was with how 'X' was split
before dispatching to bplapply() and affected both SerialParam and
SnowParam. Now fixed in release (1.2.21) and devel (1.3.52).
Valerie
- Forwarded Message -
From: "
el (1.3.52).
Valerie
- Forwarded Message -
From: "Robert Castelo"
To: bioc-devel@r-project.org
Sent: Wednesday, September 2, 2015 8:12:33 AM
Subject: [Bioc-devel] BiocParallel::bpvec() and DNAStringSet objects, problem
hi,
I have encountered a problem when using the bpvec
elo"
> To: bioc-devel@r-project.org
> Sent: Wednesday, September 2, 2015 8:12:33 AM
> Subject: [Bioc-devel] BiocParallel::bpvec() and DNAStringSet objects, problem
>
> hi,
>
> I have encountered a problem when using the bpvec() function from the
> BiocParallel p
hi,
I have encountered a problem when using the bpvec() function from the
BiocParallel package with DNAStringSet objects and the "SerialParam"
backend:
library(Biostrings)
library(BiocParallel)
## all correct when using the multicore backend
bpvec(X=DNAStringSet(c("AC", "GT")), FUN=as.char
Hi Leo,
Thanks for reporting this. It was missed by the build system (and
myself!) because the package was already installed. Should be fixed in
1.1.16.
futile.logger (and others) were moved to Suggests to lighten the load of
the NAMESPACE. Logging isn't technically turned on until the clust
Hi,
I noticed that BiocParallel::SnowParam() changed. It now uses the
futile.logger package, but it's only suggested by BiocParallel as seen here
https://github.com/Bioconductor/BiocParallel/blob/master/DESCRIPTION#L19
This leads to some errors as shown at
https://travis-ci.org/lcolladotor/derfind
On Thu, Nov 20, 2014 at 12:17 PM, Thomas Girke wrote:
> Hi Valerie,
>
> Excellent. In addition to collecting log outputs, I have a few more
> suggestions that may be worth considering:
>
> - Collecting the results form parallel computing tasks directly in an R
> object is a great convenience, w
Hi Valerie,
Excellent. In addition to collecting log outputs, I have a few more
suggestions that may be worth considering:
- Collecting the results form parallel computing tasks directly in an R
object is a great convenience, which I like a lot. However, in the
context of slow computations t
Hi Valerie, Michel and others,
Finally, I freed up some time to revisit this problem. As it turns out,
it is related to the use of a module system on our cluster. If I add in
the template file for Torque (torque.tmpl) an explicit module load line
for the specific R version, I am using on the mas
Hi Michel,
In BiocParallel 0.99.24 .convertToSimpleError() now checks for NULL and
converts to NA_character_.
I'm testing with BatchJobs 1.4, BiocParallel 0.99.24 and SLURM. I'm
still not getting an informative error message:
xx <- bplapply(1:2, FUN)
SubmitJobs |++
This was a bug in BatchJobs::waitForJobs(). We now throw an error if
jobs "disappear" due to a faulty template file. I'd appreciate if you
could confirm that this is now correctly catched and handled on your
system. I furthermore suggest to replace NULL with NA_character_ in
.convertToSimpleError()
2014-09-23 23:41 GMT+02:00 Valerie Obenchain :
> Michel Lang (cc'd) implemented BatchJobs in BiocParallel. I'd like to get
> his opinion on how he wants to handle this type of error.
> Michel, let me know if you need more details, I can send another example
> off-line.
If the cluster is misconfigu
Hi Valerie,
Thanks for looking into this.
Yes, if I include the bogus 'MYR' in *.tmpl then I am getting the same
error in R-release as well.
To double-check whether it is related to some nodes on our cluster (ours
has different node architectures and the IB interconnect can be flaky at
times),
Hi,
Martin and I looked into this a bit. It looks like a problem with
handling an 'undefined error' returned from a worker (i.e., job did not
run). When there is a problem executing the tmpl script no error message
is sent back. The NULL is coerced to simpleError and becomes a problem
downstr
Hi Thomas,
Just wanted to let you know I saw this and am looking into it.
Valerie
On 09/20/2014 02:54 PM, Thomas Girke wrote:
Hi Martin, Micheal and Vincent,
If I run the following code, with the release version of BiocParallel then it
works (took me some time to actually realize that), but w
Hi Martin, Micheal and Vincent,
If I run the following code, with the release version of BiocParallel then it
works (took me some time to actually realize that), but with the development
version I am getting an error shown after the test code below. If I run the
same test with BatchJobs from the
Just a note: the foreach package has solved this by providing a
"nesting" operator, which effectively converts multiple nested foreach
loops into one big one:
http://cran.r-project.org/web/packages/foreach/vignettes/nested.pdf
On Thu 14 Nov 2013 09:24:29 AM PST, Michael Lawrence wrote:
I like
I like the general idea of having iterators; was just checking out the
itertools package after not having looked at it for a while. I could see
having a BiocIterators package, and a bpiterate(iterator, FUN, ...,
BPPARAM). My suggestion was simpler though. Right now, bpmapply runs a
single job per i
We use a design iterator in BatchExperiments::makeDesign for a cartesian
product. I found a old version of designIterator (cf. <
https://github.com/tudo-r/BatchExperiments/blob/master/R/designs.R>) w/o
the optional data.frame input which is easier to read: <
https://gist.github.com/mllg/7469844>.
Something could go into BatchJobs, but it would be nice to have abstract
support for it at the level of BiocParallel.
On Thu, Nov 14, 2013 at 6:32 AM, Vincent Carey
wrote:
> Streamer package has DAGTeam/DAGParam components that I believe are
> relevant.
> An abstraction of the reduction plan for
Streamer package has DAGTeam/DAGParam components that I believe are
relevant.
An abstraction of the reduction plan for a parallelized task would seem to
have a natural
home in BatchJobs.
On Thu, Nov 14, 2013 at 8:15 AM, Michael Lawrence wrote:
> Hi guys,
>
> We often need to iterate over the ca
Hi guys,
We often need to iterate over the cartesian product of two dimensions, like
sample X chromosome. This is preferable to nested iteration, which is
complicated. I've been using expand.grid and bpmapply for this, but it
seems like this could be made easier. Like bpmapply could gain a CARTESI
On 11/04/2013 11:34 AM, Michael Lawrence wrote:
The dynamic nature of R limits the extent of these checks. But as Ryan has
noted, a simple sanity check goes a long way. If what he has done could be
extended to the rest of the search path (people always forget to attach
packages), I think we've hi
The 'foreach' framework does this sort of analysis using codetools at
least in part. You may be able to build on what they have.
luke
On Mon, 4 Nov 2013, Ryan wrote:
On 11/4/13, 11:05 AM, Gabriel Becker wrote:
As a side note, I'm not sure that existence of a symbol is sufficient (it
certainl
On 11/4/13, 11:05 AM, Gabriel Becker wrote:
As a side note, I'm not sure that existence of a symbol is sufficient (it
certainly is necessary). What about situations where the symbol exists but
is stale compared to the value in the parent? Are we sure that can never
happen?
I think this is a diff
Gabriel,
Thanks for the clarification. I was avoiding depending on CodeDepends
because I'm fairly certain that a BioC package can't depend on a package
that isn't in either CRAN or Bioconductor. Since you point out that the
librarySymbols code doesn't depend on any other part of the package, I
Ryan,
I agree that in some sense it is a different problem, but my point is with
a different approach we can easily answer both. The code I posted returns a
named character vector of symbol names with package name being the name.
This makes it a trivial lookup to determine both a) what symbols ar
The code that I wrote intentionally avoids checking for package variables,
since I consider that a separate problem. Package variables can be provided
to the child by leading the package, whereas user-defined variables must be
serialized in the parent and sent to the child.
I think I could fairly
Weird, I guess it needs to be logged in or something. I don't know if the
issue is that its in a non-master branch or waht. The repo is fully public
and the forCRAN_0.3.5 in branch definitely exists on github.
I started chrome (where I'm not logged into github) and got the same 404
error but afte
The dynamic nature of R limits the extent of these checks. But as Ryan has
noted, a simple sanity check goes a long way. If what he has done could be
extended to the rest of the search path (people always forget to attach
packages), I think we've hit the 80% with 20%. Got a 404 on that URL btw.
Mi
Hey guys,
Here is code that I have written which resolves library names into a full
list of symbols:
https://github.com/duncantl/CodeDepends/blob/forCRAN_0.3.5/R/librarySymbols.RNote
this does not require that the packages actually be loaded at the time
of the check, and does not load them (or rat
You might want to consider using Recall() for recursion which should solve
this. Determining the required variables using heuristics as codetools will
probably lead to some confusion when using functions which include calls
to, e.g., with():
f = function() {
with(iris, Sepal.Length + Sepal.Width
Actually, the check that I proposed is only supposed to check for usage
of user-defined variables, not variables from packages. Truthfully,
though, I guess I'm not the right person to work on this, since in
practice I use forked processes for the vast majority of my inside-R
parallelization, so
Ok, here is my attempt at a function to get the list of user-defined
free variables that a function refers to:
https://gist.github.com/DarwinAwardWinner/7298557
Is uses codetools, so it is subject to the limitations of that package,
but for simple examples, it successfully detects when a funct
Ryan (et al),
FYI:
> f
function() {
x = rnorm(x)
x
}
> findGlobals(f)
[1] "=" "{" "rnorm"
"x" should be in the list of globals but it isn't.
~G
> sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: x86_64-pc-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3]
Looking at the codetools package, I think "findGlobals" is basically
exactly what we want here, right? As you say, there are necessarily
limitations due to R being a dynamic language, but the goal is to catch
common errors, not stop people from tricking the check.
I think I'll try to code somet
I guess all we need to do is to detect whether a function would try to
access a free variable in the user's workspace, and warn/error if so.
It looks like CodeDepends could do that. I could try to come up with an
implementation. I guess we would add CodeDepends as an optional
dependency for Bio
Henrik,
See https://github.com/duncantl/CodeDepends (as used by used by
https://github.com/gmbecker/RCacheSuite). It will identify necessarily
defined symbols (input variables) for code that is not doing certain tricks
(eg get(), mixing data.frame columns and gobal variables in formulas, etc ).
T
Another potential easy step we can do is that if FUN function in the
user's workspace, we automatically export that function under the same
name in the children. This would make recursive functions just work, but
it might be a bit too magical.
On 11/3/13, 2:38 PM, Ryan wrote:
Here's an easy th
Here's an easy thing we can add to BiocParallel in the short term. The
following code defines a wrapper function "withBPExtraErrorText" that
simply appends an additional message to the end of any error that looks
like it is about a missing variable. We could wrap every evaluation in
a similar t
On Sun, Nov 3, 2013 at 1:29 PM, Michael Lawrence
wrote:
> An analog to clusterExport is a good idea. To make it even easier, we could
> have a dynamic environment based on object tables that would catch missing
> symbols and download them from the parent thread. But maybe there's some
> benefit to
An analog to clusterExport is a good idea. To make it even easier, we could
have a dynamic environment based on object tables that would catch missing
symbols and download them from the parent thread. But maybe there's some
benefit to being explicit?
Michael
On Sun, Nov 3, 2013 at 12:39 PM, Henr
Hi,
in BiocParallel, is there a suggested (or planned) best standards for
making *locally* assigned variables (e.g. functions) available to the
applied function when it runs in a separate R process (which will be
the most common use case)? I understand that avoid local variables
should be avoided
Yep, Michael will send a pull request within the next few weeks.
Michael
On Tue, Sep 3, 2013 at 6:07 AM, Hahne, Florian
wrote:
> Great, thanks for the feedback. I will give it a try asap.
>
> Am Sep 3, 2013 um 15:03 schrieb "Martin Morgan" :
>
> > On 09/03/2013 05:25 AM, Hahne, Florian wrote:
>
Great, thanks for the feedback. I will give it a try asap.
Am Sep 3, 2013 um 15:03 schrieb "Martin Morgan" :
> On 09/03/2013 05:25 AM, Hahne, Florian wrote:
>> Hi List, Martin,
>> I just wanted to quickly ask about the status of the BiocParallel package and
>> the cluster support in particular. I
On 09/03/2013 05:25 AM, Hahne, Florian wrote:
Hi List, Martin,
I just wanted to quickly ask about the status of the BiocParallel package and
the cluster support in particular. Is this project finished? And are there plans
to having BiocParallel as a proper package again, or will it remain a GIT
Hi List, Martin,
I just wanted to quickly ask about the status of the BiocParallel package and
the cluster support in particular. Is this project finished? And are there
plans to having BiocParallel as a proper package again, or will it remain a GIT
project?
Florian
[[alternative HTML v
Thanks for the info, the package now installs. Looks pretty cool. Attached
is my first hack at this from a couple of months ago. In a way it is very
similar to what you guys are doing, only that It predates the times of a
formal registry for parallel backends. I just tried to make it look like a
cl
Hi Florian,
Yes you're absolutely right. The fork currently depends on some
functions which are not yet included in the CRAN build. For now you
can get the latest development version on
http://batchjobs.googlecode.com. We'll upload a new version of
BatchJobs soon. I've documented this as an issue
This slipped under my radar, sorry. Guess there already is some
considerable work going on right now to bring queued clusters closer into
the Bioconductor world. If my first attempt on this is still of any help I
am happy to share it, of course. Just let me know.
Michael: I tried to install your Bi
Hi Henrik,
Sorry for the late response. Suggestions and feedback are always
welcome. I just forgot to enable the issue tracker (now enabled).
For prototyping I usually use Interactive/Multicore, but I'll
regularly test on our local clusters which use Torque or Slurm,
respectively.
Michel
2013/6
We're regularly running BatchJobs itself on an LSF cluster. Works great.
On Thu, Jun 6, 2013 at 5:48 PM, Henrik Bengtsson wrote:
> Great - this looks promising already.
>
> What's your test system(s), beyond standard SSH and multicore
> clusters? I'm on a Torque/PBS system.
>
> I'm happy to tes
Great - this looks promising already.
What's your test system(s), beyond standard SSH and multicore
clusters? I'm on a Torque/PBS system.
I'm happy to test, give feedback etc. I don't see an 'Issues' tab on
the GitHub page. Michel, how do you prefer to get feedback?
/Henrik
On Thu, Jun 6, 2
And here is the on-going development of the backend:
https://github.com/mllg/BiocParallel/tree/batchjobs
Not sure how well it's been tested.
Kudos to Michel Lang for making so much progress so quickly.
Michael
On Thu, Jun 6, 2013 at 1:59 PM, Dan Tenenbaum wrote:
> On Thu, Jun 6, 2013 at 1:56
On Thu, Jun 6, 2013 at 1:56 PM, Henrik Bengtsson wrote:
> Hi, I'd like to pick up the discussion on a BatchJobs backend for
> BiocParallel where it was left back in Dec 2012 (Bioc-devel thread
> 'BiocParallel'
> [https://stat.ethz.ch/pipermail/bioc-devel/2012-December/003918.html]).
>
> Florian,
Hi, I'd like to pick up the discussion on a BatchJobs backend for
BiocParallel where it was left back in Dec 2012 (Bioc-devel thread
'BiocParallel'
[https://stat.ethz.ch/pipermail/bioc-devel/2012-December/003918.html]).
Florian, would you mind sharing your BatchJobs backend code? Is it
independe
By the way, all my work on BiocParallel is going to end up here:
https://github.com/DarwinAwardWinner/BiocParallel
If you want to read through the multicore-only pvectorize, it is here:
https://github.com/DarwinAwardWinner/BiocParallel/blob/a3699cf/R/pvectorize.R
It's a little more than one l
On Tue, Dec 4, 2012 at 12:47 PM, Ryan C. Thompson wrote:
> One issue that I see is that for some kinds of parallel backends, there
> may not be any way for "bpworkers" to return something meaningful. For
> example, a backend that submits jobs to a large cluster may not know
> exactly how many node
One issue that I see is that for some kinds of parallel backends, there
may not be any way for "bpworkers" to return something meaningful. For
example, a backend that submits jobs to a large cluster may not know
exactly how many nodes are in the cluster, and in any case returning the
total numb
Thanks.
On Tue, Dec 4, 2012 at 3:47 AM, Vincent Carey
wrote:
> I have been booked up so no chance to deploy but I do have access to SGE and
> LSF so will try and will report ASAP.
...and I'll try it out on PBS (... but I most likely won't have time
to do this until the end of the year).
Henrik
On Tue 04 Dec 2012 11:31:59 AM PST, Michael Lawrence wrote:
The name "pvec" is not very intuitive. What about "bpchunk"? And since the
function passed to bpvectorize is already vectorized, maybe bpvectorize
should be bparallelize? I know everyone has different
intuitions/preferences when it comes
Looks like great progress has been made. Here are some thoughts:
The *Params objects seem to have two roles: specifying the desired
resources and indicating the scheduler (the thing that actually executes
the jobs). Maybe it would be beneficial to have separate abstractions for
those two things. F
I have been booked up so no chance to deploy but I do have access to SGE
and LSF so will try and will report ASAP.
On Tue, Dec 4, 2012 at 4:08 AM, Hahne, Florian
wrote:
> Hi Henrik,
> I have now come up now with a relatively generic version of this
> SGEcluster approach. It does indeed use BatchJ
Hi Henrik,
I have now come up now with a relatively generic version of this
SGEcluster approach. It does indeed use BatchJobs under the hood and
should thus support all available cluster queues, assuming that the
necessary batchJobs routines are available. I could only test this on our
SGE cluster,
Picking up this thread in lack of other places (= were should
BiocParallel be discussed?)
I saw Martin's updates on the BiocParallel - great. Florian's SGE
scheduler was also mentioned; is that one built on top of BatchJobs?
If so I'd be interested in looking into that/generalizing that to work
w
Bioc Developers --
BiocParallel generated quite a bit of discussion, so I'm providing a brief
update. Version 0.0.5 is available to R-devel users via biocLite; it's in svn
https://hedgehog.fhcrc.org/bioconductor/trunk/madman/Rpacks/BiocParallel
and github
https://github.com/Bioconductor/
1 - 100 of 133 matches
Mail list logo
