Here is a sample of the output, It almost instantly uses 2GB and then starts using VMem. This is probably the right suggestion but it's another thing to install
> It's probably also worth being aware of guppy's heapy stuff: http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html>I find it quite nice to have the following to get a quick point-in-time estimate of my app's memory usage: print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024)) 19 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_nm_rep1.txt 3_veg_nm_rep1.txt has 228484 rows of data 3_veg_nm_rep1.txt has 0 rows of masked data 3_veg_nm_rep1.txt has 141 rows of outliers 3_veg_nm_rep1.txt has 0 modified rows of data 280bytes of memory used for 3_veg_nm_rep1.txt 20 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_nm_rep2.txt 3_veg_nm_rep2.txt has 228484 rows of data 3_veg_nm_rep2.txt has 0 rows of masked data 3_veg_nm_rep2.txt has 119 rows of outliers 3_veg_nm_rep2.txt has 0 modified rows of data 280bytes of memory used for 3_veg_nm_rep2.txt 21 files of 43 using 3352 bytes of memory Loading into memory: 3_veg_phd_rep1.txt 3_veg_phd_rep1.txt has 228484 rows of data 3_veg_phd_rep1.txt has 0 rows of masked data 3_veg_phd_rep1.txt has 63 rows of outliers 3_veg_phd_rep1.txt has 0 modified rows of data 280bytes of memory used for 3_veg_phd_rep1.txt 22 files of 43 using 6424 bytes of memory Loading into memory: 3g_c285-11.txt 3g_c285-11.txt has 228484 rows of data 3g_c285-11.txt has 0 rows of masked data 3g_c285-11.txt has 65 rows of outliers 3g_c285-11.txt has 0 modified rows of data 280bytes of memory used for 3g_c285-11.txt 23 files of 43 using 6424 bytes of memory Loading into memory: 3g_c285-42.txt 3g_c285-42.txt has 228484 rows of data 3g_c285-42.txt has 0 rows of masked data 3g_c285-42.txt has 27 rows of outliers 3g_c285-42.txt has 0 modified rows of data 280bytes of memory used for 3g_c285-42.txt 24 files of 43 using 6424 bytes of memory Loading into memory: A6AF.txt A6AF.txt has 228484 rows of data A6AF.txt has 0 rows of masked data A6AF.txt has 36 rows of outliers A6AF.txt has 0 modified rows of data 280bytes of memory used for A6AF.txt 25 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep1.txt Grigg_3026_rep1.txt has 228484 rows of data Grigg_3026_rep1.txt has 0 rows of masked data Grigg_3026_rep1.txt has 949 rows of outliers Grigg_3026_rep1.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep1.txt 26 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep2.txt Grigg_3026_rep2.txt has 228484 rows of data Grigg_3026_rep2.txt has 0 rows of masked data Grigg_3026_rep2.txt has 361 rows of outliers Grigg_3026_rep2.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep2.txt 27 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3026_rep3_both.txt Grigg_3026_rep3_both.txt has 228484 rows of data Grigg_3026_rep3_both.txt has 0 rows of masked data Grigg_3026_rep3_both.txt has 41 rows of outliers Grigg_3026_rep3_both.txt has 0 modified rows of data 280bytes of memory used for Grigg_3026_rep3_both.txt 28 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3131_rep1.txt Grigg_3131_rep1.txt has 228484 rows of data Grigg_3131_rep1.txt has 0 rows of masked data Grigg_3131_rep1.txt has 537 rows of outliers Grigg_3131_rep1.txt has 0 modified rows of data 280bytes of memory used for Grigg_3131_rep1.txt 29 files of 43 using 6424 bytes of memory Loading into memory: Grigg_3131_rep2.txt Grigg_3131_rep2.txt has 228484 rows of data Grigg_3131_rep2.txt has 0 rows of masked data Grigg_3131_rep2.txt has 238 rows of outliers Grigg_3131_rep2.txt has 0 modified rows of data 280bytes of memory used for Grigg_3131_rep2.txt 30 files of 43 using 6424 bytes of memory *Vincent Davis 720-301-3003 * vinc...@vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis> On Sat, Feb 20, 2010 at 7:40 PM, sstein...@gmail.com <sstein...@gmail.com>wrote: > > On Feb 20, 2010, at 9:21 PM, Vincent Davis wrote: > > See this article for some more info about the reported sizes of things: >> http://www.doughellmann.com/PyMOTW/sys/limits.html > > > <http://www.doughellmann.com/PyMOTW/sys/limits.html>Nice article but I > must have missed something useful to my current issue. Do I get any hints? > > > Oh, sorry, there was the part about getsizeof() not including attributes > unless the class supplies a __sizeof__ method and a comment at the bottom: > > It's probably also worth being aware of guppy's heapy stuff: > http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html> > > I find it quite nice to have the following to get a quick point-in-time > estimate of my app's memory usage: > > print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024)) > > Put a few of those in at various places in the collection process and you > should see where you're taking a substantial hit on each pass. > > If I remember those numbers right, 5mb != 2.5gb so something along the way > is doing something very strange. > > S > >
-- http://mail.python.org/mailman/listinfo/python-list
