This is along the lines of what I suggested on the board phone call. If there is already a C++ library like Armadillo for doing the heavy lifting, it would be easy to implement an R-level abstraction on top of it, just as was done for HDF5, bigMemory, etc.
On Fri, Mar 3, 2017 at 8:32 AM, McDavid, Andrew < andrew_mcda...@urmc.rochester.edu> wrote: > On C++, Armadillo can be passed a a pointer to memory for the backing > store of its objects, so can use memory mapping. On the R side, package > bigmemory provides R access and initialization of memory-mapped arrays. > See https://www.r-bloggers.com/using-rcpparmadillo-with-bigmemory/. This > doesn’t provide language or platform interchange of the backing store, but > would be an easy-ish solution. > > On Mar 3, 2017, at 10:23 AM, bioc-devel-requ...@r-project.org<mailto: > bioc-devel-requ...@r-project.org> wrote: > > Some comment on Aaron's stuff > > One possibility for doing things like this is if your code can be done in > C++ using a subset of rows or columns. That can sometimes give the > necessary speed up. What I mean is this > > Say you can safely process 1000 cells (not matrix cells, but biological > cells, aka columns) at a time in RAM > > iterate in R: > get chunk i containing 1000 cells from the backend data storage > do something on this sub matrix where everything is in a normal matrix > and you just use C++ > write results out to whatever backend you're using > > Then, with a million cells you iterate over 1000 chunks in R. And you > don't need to "touch" the full dataset which can be stored on an arbitrary > backend. And this approach could be run even (potentially) with different > chunks on different nodes. > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
