Yes, bigmemoryExtras::BigMatrix and genoset::RleDataFrame() are good tricks
for reducing the size of your eSets and SummarizedExperiments. Both object
types can go into assayData or assays. In fact, that's what they were
designed for.
At Genentech, we use these for our 2.5e6 x 1e3 rectangular data from
Illumina SNP arrays. We typically have ~6 such rectangular objects in one
eSet. With a mix of BigMatrix object for point estimates and RleDataFrames
for segmented data, readRDS times are quite reasonable.
Pete
____________________
Peter M. Haverty, Ph.D.
Genentech, Inc.
phave...@gene.com
On Fri, Sep 18, 2015 at 1:56 PM, Tim Triche, Jr. <tim.tri...@gmail.com>
wrote:
> bigmemoryExtras (Peter Haverty's extensions to bigMemory/bigMatrix) can be
> handy for this, as it works well as a backend, especially if you go about
> splitting by chromosome as for CNV segmentation, DMR finding, etc. It's
> not as seamless as one might like, but it's the closest thing I've found.
>
> SciDb tries to implement a similar API, but for a distributed version of
> this where the data itself is in a columnar database and served on demand.
> I tried getting that up and running as a SummarizedExperiment backend, but
> did not succeed. I have previously shoveled all of the TCGA 450k data into
> one 7,000+ column bigMatrix which serializes to about 14GB on disk.
>
> If you have any replicates in your 700+ samples, it's a good idea to keep
> their SNP calls in metadata(yourSE), although if you change names it needs
> to propagate into the dependent metadata. This is why I started monkeying
> around with linkedExperiments where those mappings are enforced; it's
> becoming more of an issue with the TARGET pediatric AML study, where there
> are numerous diagnosis-remission-relapse trios whose identity I wish to
> verify periodically. The SNPs on the 450k array are great for this
> purpose, but minfi doesn't really have a slot for them per se, so live in
> metadata().
>
>
> --t
>
> On Fri, Sep 18, 2015 at 1:29 PM, Vincent Carey <st...@channing.harvard.edu
> >
> wrote:
>
> > i am dealing with ~700 450k arrays
> >
> > they are derived from one study, so it makes sense to think of
> >
> > them holistically.
> >
> > both the load time and the memory consumption are not satisfactory.
> >
> > has anyone worked on an object type that implements the rangedSE API but
> > has
> >
> > the assay data out of memory?
> >
> > > unix.time(load("wbmse.rda"))
> >
> > user system elapsed
> >
> > 30.131 2.396 61.036
> >
> > > object.size(wbmse)
> >
> > 124031032 bytes
> >
> > > dim(wbmse)
> >
> > [1] 485577 690
> >
> > > object.size(assays(wbmse))
> >
> > 2680430992 bytes
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
