Well, you could always try a different method of multiplication. A common trick would be to use block multiplication. http://en.wikipedia.org/wiki/Block_matrix
On Sun, Jan 16, 2011 at 4:30 PM, <bangpypers-requ...@python.org> wrote: > Send BangPypers mailing list submissions to > bangpypers@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/bangpypers > or, via email, send a message with subject or body 'help' to > bangpypers-requ...@python.org > > You can reach the person managing the list at > bangpypers-ow...@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BangPypers digest..." > > > Today's Topics: > > 1. Multiplying very large matrices (kunal ghosh) > 2. Re: Multiplying very large matrices (Santosh Rajan) > 3. Re: Multiplying very large matrices (kunal ghosh) > 4. Re: Multiplying very large matrices (kunal ghosh) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 15 Jan 2011 22:11:21 +0530 > From: kunal ghosh <kunal...@gmail.com> > To: Bangalore Python Users Group - India <bangpypers@python.org> > Subject: [BangPypers] Multiplying very large matrices > Message-ID: > > <aanlktin0twkncs7wabq581nmla2qo+hotjbbds94+...@mail.gmail.com<aanlktin0twkncs7wabq581nmla2qo%2bhotjbbds94%2b...@mail.gmail.com> > > > Content-Type: text/plain; charset=ISO-8859-1 > > Hi all, > while implementing Locality Preserving Projections , > at one point i have to perform X L X.transpose() > these matrices are large (32256 x 32256) so i get "out of memory" error. > > I assume, as the dataset gets larger one would come across this problem , > how would > one go about solving this ? Is there a common trick that is used to deal > with such problems ? > Or the workstation calculating these problems needs to have HUGE amounts > of > physical memory ? > > I am using python and numpy / scipy > > -- > regards > ------- > Kunal Ghosh > Dept of Computer Sc. & Engineering. > Sir MVIT > Bangalore,India > > permalink: member.acm.org/~kunal.t2 > Blog:kunalghosh.wordpress.com > Website:www.kunalghosh.net46.net > > > ------------------------------ > > Message: 2 > Date: Sat, 15 Jan 2011 23:12:01 +0530 > From: Santosh Rajan <santra...@gmail.com> > To: Bangalore Python Users Group - India <bangpypers@python.org> > Subject: Re: [BangPypers] Multiplying very large matrices > Message-ID: > <aanlktikq-ywrjjfdkgfw_t8_8-r2o0wxy9dmuxbfz...@mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hope this helps > http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices > > On Sat, Jan 15, 2011 at 10:11 PM, kunal ghosh <kunal...@gmail.com> wrote: > > Hi all, > > while implementing Locality Preserving Projections , > > at one point i have to perform X L X.transpose() > > these matrices are large (32256 x 32256) so i get "out of memory" error. > > > > I assume, as the dataset gets larger one would come across this problem , > > how would > > one go about solving this ? Is there a common trick that is used to deal > > with such problems ? > > Or the workstation calculating these problems needs to have HUGE ?amounts > of > > physical memory ? > > > > I am using python and numpy / scipy > > > > -- > > regards > > ------- > > Kunal Ghosh > > Dept of Computer Sc. & Engineering. > > Sir MVIT > > Bangalore,India > > > > permalink: member.acm.org/~kunal.t2 > > Blog:kunalghosh.wordpress.com > > Website:www.kunalghosh.net46.net > > _______________________________________________ > > BangPypers mailing list > > BangPypers@python.org > > http://mail.python.org/mailman/listinfo/bangpypers > > > > > > -- > http://about.me/santosh.rajan > > > ------------------------------ > > Message: 3 > Date: Sat, 15 Jan 2011 23:34:41 +0530 > From: kunal ghosh <kunal...@gmail.com> > To: Bangalore Python Users Group - India <bangpypers@python.org> > Subject: Re: [BangPypers] Multiplying very large matrices > Message-ID: > <AANLkTinVmsKaWNx=rxyabx5uosotsvbb3ewfpn5o4...@mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Thanks Santosh , > > This stack overflow thread indeed discusses the exact same problem i have. > Wonder how i missed it :) in my preliminary searches. > > thanks again ! > > On Sat, Jan 15, 2011 at 11:12 PM, Santosh Rajan <santra...@gmail.com> > wrote: > > > Hope this helps > > > http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices > > > > On Sat, Jan 15, 2011 at 10:11 PM, kunal ghosh <kunal...@gmail.com> > wrote: > > > Hi all, > > > while implementing Locality Preserving Projections , > > > at one point i have to perform X L X.transpose() > > > these matrices are large (32256 x 32256) so i get "out of memory" > error. > > > > > > I assume, as the dataset gets larger one would come across this problem > , > > > how would > > > one go about solving this ? Is there a common trick that is used to > deal > > > with such problems ? > > > Or the workstation calculating these problems needs to have HUGE > amounts > > of > > > physical memory ? > > > > > > I am using python and numpy / scipy > > > > > > -- > > > regards > > > ------- > > > Kunal Ghosh > > > Dept of Computer Sc. & Engineering. > > > Sir MVIT > > > Bangalore,India > > > > > > permalink: member.acm.org/~kunal.t2 > > > Blog:kunalghosh.wordpress.com > > > Website:www.kunalghosh.net46.net > > > _______________________________________________ > > > BangPypers mailing list > > > BangPypers@python.org > > > http://mail.python.org/mailman/listinfo/bangpypers > > > > > > > > > > > -- > > http://about.me/santosh.rajan > > _______________________________________________ > > BangPypers mailing list > > BangPypers@python.org > > http://mail.python.org/mailman/listinfo/bangpypers > > > > > > -- > regards > ------- > Kunal Ghosh > Dept of Computer Sc. & Engineering. > Sir MVIT > Bangalore,India > > permalink: member.acm.org/~kunal.t2 > Blog:kunalghosh.wordpress.com > Website:www.kunalghosh.net46.net > > > ------------------------------ > > Message: 4 > Date: Sun, 16 Jan 2011 12:32:01 +0530 > From: kunal ghosh <kunal...@gmail.com> > To: Bangalore Python Users Group - India <bangpypers@python.org> > Subject: Re: [BangPypers] Multiplying very large matrices > Message-ID: > > <aanlktim5yij63orhntjmhni4xaug+bx2mfx2redva...@mail.gmail.com<aanlktim5yij63orhntjmhni4xaug%2bbx2mfx2redva...@mail.gmail.com> > > > Content-Type: text/plain; charset=ISO-8859-1 > > Hi all, > I found numpy.memmap > to be very suitable when matrices larger than the physical memory are > required. > > 1. included in standard numpy installation > 2. very low learning curve. > > pyTables seems to be more suitable but , i somehow found the learning curve > too steep . Also pyTables needs lot of initializations before anything can > be done with it > as compared to memmap. > > The above reason made me use memmap over pyTable. > > On Sat, Jan 15, 2011 at 11:34 PM, kunal ghosh <kunal...@gmail.com> wrote: > > > Thanks Santosh , > > > > This stack overflow thread indeed discusses the exact same problem i > have. > > Wonder how i missed it :) in my preliminary searches. > > > > thanks again ! > > > > > > On Sat, Jan 15, 2011 at 11:12 PM, Santosh Rajan <santra...@gmail.com > >wrote: > > > >> Hope this helps > >> > >> > http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices > >> > >> On Sat, Jan 15, 2011 at 10:11 PM, kunal ghosh <kunal...@gmail.com> > wrote: > >> > Hi all, > >> > while implementing Locality Preserving Projections , > >> > at one point i have to perform X L X.transpose() > >> > these matrices are large (32256 x 32256) so i get "out of memory" > error. > >> > > >> > I assume, as the dataset gets larger one would come across this > problem > >> , > >> > how would > >> > one go about solving this ? Is there a common trick that is used to > deal > >> > with such problems ? > >> > Or the workstation calculating these problems needs to have HUGE > >> amounts of > >> > physical memory ? > >> > > >> > I am using python and numpy / scipy > >> > > >> > -- > >> > regards > >> > ------- > >> > Kunal Ghosh > >> > Dept of Computer Sc. & Engineering. > >> > Sir MVIT > >> > Bangalore,India > >> > > >> > permalink: member.acm.org/~kunal.t2 > >> > Blog:kunalghosh.wordpress.com > >> > Website:www.kunalghosh.net46.net > >> > _______________________________________________ > >> > BangPypers mailing list > >> > BangPypers@python.org > >> > http://mail.python.org/mailman/listinfo/bangpypers > >> > > >> > >> > >> > >> -- > >> http://about.me/santosh.rajan > >> _______________________________________________ > >> BangPypers mailing list > >> BangPypers@python.org > >> http://mail.python.org/mailman/listinfo/bangpypers > >> > > > > > > > > -- > > regards > > ------- > > Kunal Ghosh > > Dept of Computer Sc. & Engineering. > > Sir MVIT > > Bangalore,India > > > > permalink: member.acm.org/~kunal.t2 > > Blog:kunalghosh.wordpress.com > > Website:www.kunalghosh.net46.net > > > > > > > > > -- > regards > ------- > Kunal Ghosh > Dept of Computer Sc. & Engineering. > Sir MVIT > Bangalore,India > > permalink: member.acm.org/~kunal.t2 > Blog:kunalghosh.wordpress.com > Website:www.kunalghosh.net46.net > > > ------------------------------ > > _______________________________________________ > BangPypers mailing list > BangPypers@python.org > http://mail.python.org/mailman/listinfo/bangpypers > > > End of BangPypers Digest, Vol 41, Issue 27 > ****************************************** > _______________________________________________ BangPypers mailing list BangPypers@python.org http://mail.python.org/mailman/listinfo/bangpypers
