Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-19 Thread Vincent Carey
On Fri, Sep 18, 2015 at 8:36 PM, Kasper Daniel Hansen < kasperdanielhan...@gmail.com> wrote: > Interesting, thanks for the pointer. > > In light of the existing (and future) work on this, may I suggest an eSet > like class, but build using the technologies in SummarizedExperiment. Ie. > a Summari

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-19 Thread Benilton Carvalho
Lawrence > > Sent: Friday, September 18, 2015 10:42 PM > > To: Peter Haverty > > Cc: Tim Triche, Jr.; bioc-devel@r-project.org > > Subject: Re: [Bioc-devel] SummarizedExperiment with alternate back end > > > > While it's useful (and often necessary) to st

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-19 Thread Morgan, Martin
n...@r-project.org] On Behalf Of > Michael Lawrence > Sent: Friday, September 18, 2015 10:42 PM > To: Peter Haverty > Cc: Tim Triche, Jr.; bioc-devel@r-project.org > Subject: Re: [Bioc-devel] SummarizedExperiment with alternate back end > > While it's useful (and often necessary)

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Ryan
For what it's worth, I've written a class which I have creatively named SubsettableListOfArrays which basically taking the "subset everything together" aspect of eSet and SummarizedExperiment and making it as generic as possible. It's basically like (non-ranged) SummarizedExperiment, except tha

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Michael Lawrence
While it's useful (and often necessary) to store the big matrices out of core, it would be convenient to store the metadata (the other components of the object) along with the matrices. Something along the lines of HDF5, but we would want to keep things abstract. Other options include GDS (for geno

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Peter Haverty
While we are on the topic, my GenoSet class will become a subclass of RangedSummarizedExperiment, rather than eSet, after this upcoming release. For this release both APIs work (colnames and sampleNames, etc.) I think the range-free SummarizedExperiment will be great. I've seen a lot of Expression

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Ryan
In the dev version, SummarizedExperiment has been split into RangedSummarizedExperiment (equivalent to the current SummarizedExperiement, with rowRanges) and SummarizedExperiment (kind of like eSet, no rowRanges). Given that eSet objects also support multiple assayData elements, I believe the n

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Kasper Daniel Hansen
Interesting, thanks for the pointer. In light of the existing (and future) work on this, may I suggest an eSet like class, but build using the technologies in SummarizedExperiment. Ie. a SummarizedExperiment without the rowRanges. I would very much like this for modern work using eSet like conta

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Vincent Carey
thanks to all, lots of potential here. On Fri, Sep 18, 2015 at 3:28 PM, Peter Haverty wrote: > Yes, bigmemoryExtras::BigMatrix and genoset::RleDataFrame() are good > tricks for reducing the size of your eSets and SummarizedExperiments. Both > object types can go into assayData or assays. In fac

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Peter Haverty
Yes, bigmemoryExtras::BigMatrix and genoset::RleDataFrame() are good tricks for reducing the size of your eSets and SummarizedExperiments. Both object types can go into assayData or assays. In fact, that's what they were designed for. At Genentech, we use these for our 2.5e6 x 1e3 rectangular dat

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Tim Triche, Jr.
bigmemoryExtras (Peter Haverty's extensions to bigMemory/bigMatrix) can be handy for this, as it works well as a backend, especially if you go about splitting by chromosome as for CNV segmentation, DMR finding, etc. It's not as seamless as one might like, but it's the closest thing I've found. S

Re: [Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Hervé Pagès
Hi Vince, This is actually on "our" TODO list for after the release. SummarizedExperiment was designed to support alternate back end for the assays slot. More details and proof of concept are in ?Assays. H. On 09/18/2015 01:29 PM, Vincent Carey wrote: i am dealing with ~700 450k arrays they a

[Bioc-devel] SummarizedExperiment with alternate back end

2015-09-18 Thread Vincent Carey
i am dealing with ~700 450k arrays they are derived from one study, so it makes sense to think of them holistically. both the load time and the memory consumption are not satisfactory. has anyone worked on an object type that implements the rangedSE API but has the assay data out of memory? >