I think it makes sense. A lot of sense. Might be useful to involve Henrik (matrixStats) as well.
Who are the players, apart from DelayedArray/DelayedMatrixStats and matter? (and some very old stuff in Biobase which should really be deprecated in favor of matrixStats). Best, Kasper On Wed, Nov 1, 2017 at 3:03 PM, Bemis, Kylie <k.be...@northeastern.edu> wrote: > Hi all, > > To continue a variant of this conversation, with the latest BioC release, > we now have quite a few packages that are implementing various > matrix-related S4 generic functions, many of them relying on matrixStats as > a template. > > I was wondering if there is any interest or intention to create a common > MatrixGenerics/ArrayGenerics package on which we can depend to import the > relevant S4 generic functions. Although BiocGeneric has a few like > ‘rowSums()’ and ‘colMeans()’, etc., there are many more that are > implemented across ‘DelayedArray', ‘DelayedMatrixStats', my own package > ‘matter', etc., including ‘apply()’, ‘rowSds()’, ‘colVars()’, and so forth. > > It would be nice to have a single package with minimal additional > dependencies (a la BiocGenerics) where we could import the various S4 > generics and avoid unwanted namespace collisions. > > Have there been any thoughts on this? > > Many thanks, > Kylie > > ~~~ > Kylie Ariel Bemis > Future Faculty Fellow > College of Computer and Information Science > Northeastern University > kuwisdelu.github.io<https://kuwisdelu.github.io> > > > > > On Mar 3, 2017, at 11:27 AM, Kasper Daniel Hansen < > kasperdanielhan...@gmail.com<mailto:kasperdanielhan...@gmail.com>> wrote: > > > > On Fri, Mar 3, 2017 at 10:22 AM, Vincent Carey <st...@channing.harvard.edu > <mailto:st...@channing.harvard.edu>> wrote: > > > On Fri, Mar 3, 2017 at 10:07 AM, Kasper Daniel Hansen < > kasperdanielhan...@gmail.com<mailto:kasperdanielhan...@gmail.com>> wrote: > Some comment on Aaron's stuff > > One possibility for doing things like this is if your code can be done in > C++ using a subset of rows or columns. That can sometimes give the > necessary speed up. What I mean is this > > Say you can safely process 1000 cells (not matrix cells, but biological > cells, aka columns) at a time in RAM > > iterate in R: > get chunk i containing 1000 cells from the backend data storage > do something on this sub matrix where everything is in a normal matrix > and you just use C++ > write results out to whatever backend you're using > > Then, with a million cells you iterate over 1000 chunks in R. And you > don't need to "touch" the full dataset which can be stored on an arbitrary > backend. > > you "touch" it, but you never ingest the whole thing at any time, is that > what you mean? > > Yes, you load the chunk into RAM and then just deal with it. > > Think of doing 10^10 linear models. If this was 10^6 I would just use > lmFit. But 10^10 doesn't fit into memory. So I load 10^7 into memory, run > lmFit, store results, redo. This is bound to be much more efficient than > loading a single row into memory and doing lm 10^10 times, because lmFit is > written to do many linear models at the same time. > > I am suggesting that this is a potential general strategy. > > > And this approach could be run even (potentially) with different chunks on > different nodes. > > that seems to me to be an important if not essential desideratum. > > what then is the role of C++? extracting a chunk? preexisting utilities? > > When I say C++ I just mean write an efficient implementation that works on > a chunk, like lmFit. It is true that anything that works on a chunk will > work on a single row/column (like lmFit) but there are possibilities for > optimization when you work at the chunk level. > > Obviously not all computations can be done chunkwise. But for those that > can, this is a strategy which is independent of the data backend. > > I wonder whether this "obviously not" needs to be rethought. Algorithms > that are implemented to work with data holistically may need > to be reexpressed so that they can succeed with chunkwise access. Is this > a new mindset needed for holist developers, or can the > effective data decompositions occur autonomously? > > Well, I would say it is obvious that not all computations can be done > chunkwise. But of course, in the limit of extremely large data, algorithms > which needs to cycle over everything no longer scale. So in that case all > practical computations can be done chunkwise, out of necessity. For single > cell right now where it is just millions of cells on the horizon people > will think that they can get "standard" holistic approaches to work (and > that is probably true). If they had a billion cells they probably wouldn't > think about that. > > Kasper > > If you need direct access to the data in the backend in C++ it will be > extremely backend dependent what is fast and how to do it. That doesn't > mean we shouldn't do it though. > > Best, > Kasper > > > > On Fri, Mar 3, 2017 at 6:47 AM, Vincent Carey <st...@channing.harvard.edu< > mailto:st...@channing.harvard.edu>> wrote: > Kylie, thanks for reminding us of matter -- I saw you speak about this at > the first Bioconductor Boston Meetup, but it > went like lightning. For developers contemplating an approach to > representing high-volume rectangular data, > where there is no dominant legacy format, it is natural to wonder whether > HDF5 would be adequate, and, > further, to wonder how to demonstrate that it is or is not dominated by > some other approach for a given set > of tasks. Should we devise a set of bioinformatic benchmark problems to > foster comparison and informed > decisionmaking? @becker.gabe: is ALTREP far enough along that one could > contemplate benchmarking with it? > > On Fri, Feb 24, 2017 at 7:08 PM, Bemis, Kylie <k.be...@northeastern.edu< > mailto:k.be...@northeastern.edu>> > wrote: > > > It’s not there yet, but I plan to expose a C++ API for my disk-backed > > matrix objects in the next version of my ‘matter’ package. > > > > It’s getting easier to interchange matter/HDF5Array/bigmemory/etc. > > objects at the R level, especially if using a frontend like DelayedArray > on > > top of them, but it would be nice to have a common C++ API that I could > > hook into as well (a la Rcpp), so new C/C++ could be re-used across > various > > backends more easily. > > > > Kylie > > > > ~~~ > > Kylie Ariel Bemis > > Future Faculty Fellow > > College of Computer and Information Science > > Northeastern University > > kuwisdelu.github.io<http://kuwisdelu.github.io/><https:// > kuwisdelu.github.io<https://kuwisdelu.github.io/>> > > > > > > > > > > On Feb 24, 2017, at 4:50 PM, Aaron Lun <a...@wehi.edu.au<mailto:alun@ > wehi.edu.au><mailto:alun@<mailto:alun@> > > wehi.edu.au<http://wehi.edu.au/>>> wrote: > > > > It's a good place to start, though it would be very handy to have a C(++) > > API that can be linked against. I'm not sure how much work that would > > entail but it would give downstream developers a lot more options. Sort > of > > like how we can link to Rhtslib, which speeds up a lot of BAM file > > processing, instead of just relying on Rsamtools. > > > > > > -Aaron > > > > ________________________________ > > From: Tim Triche, Jr. <tim.tri...@gmail.com<mailto:tim.tri...@gmail.com > ><mailto:tim.tri...@gmail.com<mailto:tim.tri...@gmail.com>>> > > Sent: Saturday, 25 February 2017 8:34:58 AM > > To: Aaron Lun > > Cc: bioc-devel@r-project.org<mailto:bioc-devel@r-project.org><mailto: > bioc-devel@r-project.org<mailto:bioc-devel@r-project.org>> > > Subject: Re: [Bioc-devel] any interest in a BiocMatrix core package? > > > > yes > > > > the DelayedArray framework that handles HDF5Array, etc. seems like the > > right choice? > > > > --t > > > > On Fri, Feb 24, 2017 at 1:26 PM, Aaron Lun <a...@wehi.edu.au<mailto: > a...@wehi.edu.au><mailto:alun@<mailto:alun@> > > wehi.edu.au<http://wehi.edu.au/>><mailto:a...@wehi.edu.au<mailto: > a...@wehi.edu.au>>> wrote: > > Hi everyone, > > > > I just attended the Human Cell Atlas meeting in Stanford, and people were > > talking about gene expression matrices for >1 million cells. If we assume > > that we can get non-zero expression profiles for ~5000 genes, we�d be > > talking about a 5000 x 1 million matrix for the raw count data. This > would > > be 20-40 GB in size, which would clearly benefit from sparse (via Matrix) > > or disk-backed representations (bigmatrix, BufferedMatrix, rhdf5, etc.). > > > > I�m wondering whether there is any appetite amongst us for making a > > consistent BioC API to handle these matrices, sort of like what > > BiocParallel does for multicore and snow. It goes without saying that the > > different matrix representations should have consistent functions at the > R > > level (rbind/cbind, etc.) but it would also be nice to have an integrated > > C/C++ API (accessible via LinkedTo). There�s many non-trivial things that > > can be done with this type of data, and it is often faster and more > memory > > efficient to do these complex operations in compiled code. > > > > I was thinking of something that you could supply any supported matrix > > representation to a registered function via .Call; the C++ constructor > > would recognise the type of matrix during class instantiation; and > > operations (row/column/random read access, also possibly various ways of > > writing a matrix) would be overloaded and behave as required for the > class. > > Only the implementation of the API would need to care about the nitty > > gritty of each representation, and we would all be free to write code > that > > actually does the interesting analytical stuff. > > > > Anyway, just throwing some thoughts out there. Any comments appreciated. > > > > Cheers, > > > > Aaron > > > > [[alternative HTML version deleted]] > > > > > > _______________________________________________ > > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org><mailto: > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org>><mailto: > > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org>> mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > > > [[alternative HTML version deleted]] > > > > _______________________________________________ > > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org> mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > > > [[alternative HTML version deleted]] > > > > _______________________________________________ > > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org> mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org> mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
