Neal Norwitz <[EMAIL PROTECTED]> wrote: > Alex Martelli wrote: > > > > So, I thought I'd turn to the "wisdom of crowds"... how would YOU guys > > go about adding to your automated regression tests one that checks that > > a certain memory leak has not recurred, as cross-platform as feasible? > > In particular, how would you code _memsize() "cross-platformly"? (I can > > easily use C rather than Python if needed, adding it as an auxiliary > > function for testing purposes to my existing extension). > > If you are doing Unix, can you use getrusage(2)?
On Unix, I could; on Linux, nope. According to man getrusage on Linux, """ The above struct was taken from BSD 4.3 Reno. Not all fields are meaningful under Linux. Right now (Linux 2.4, 2.6) only the fields ru_utime, ru_stime, ru_minflt, ru_majflt, and ru_nswap are maintained. """ and indeed the memory-usage parts are zero. > >>> import resource > >>> r = resource.getrusage(resource.RUSAGE_SELF) > >>> print r[2:5] > > I get zeroes on my gentoo amd64 box. Not sure why. I thought maybe it > was Python, but C gives the same results. Yep -- at least, on Linux, this misbehavior is clearly documented in the manpage; on Darwin, aka MacOSX, you _also_ get zeros but there is no indication in the manpage leading you to expect that. Unfortunately I don't have any "real Unix" box around -- only Linux and Darwin... I could try booting up OpenBSD again to check that it works there, but given that I know it doesn't work under the most widespread unixoid systems, it wouldn't be much use anyway, sigh. > Another possibiity is to call sbrk(0) which should return the top of > the heap. You could then return this value and check it. It requires > a tiny C module, but should be easy and work on most unixes. You can As I said, I'm looking for leaks in a C-coded module, so it's no problem to add some auxiliary C code to that module to help test it -- unfortunately, this approach doesn't work, see below... > determine direction heap grows by comparing it with id(0) which should > have been allocated early in the interpreters life. > > I realize this isn't perfect as memory becomes fragmented, but might > work. Since 2.3 and beyond use pymalloc, fragmentation may not be much > of an issue. As memory is allocated in a big hunk, then doled out as > necessary. But exactly because of that, sbrk(0) doesn't mean much. Consider the tiny extension which I've just uploaded to http://www.aleax.it/Python/memtry.c -- it essentially exposes a type that does malloc when constructed and free when freed, and a function sbrk0 which returns sbrk(0). What I see on my MacOSX 10.4, Python 2.4.1, gcc 4.1, is (with a little auxiliary memi.py module that does from memtry import * import os def memsiz(): return int(os.popen('ps -p %d -o vsz|tail -1' % os.getpid()).read()) )...: Helen:~/memtry alex$ python -ic 'import memi' >>> memi.memsiz() 35824 >>> memi.sbrk0() 16809984 >>> a=memi.mem(999999) >>> memi.sbrk0() 16809984 >>> memi.memsiz() 40900 See? While the process's memory size grows as expected (by 500+ "units" when allocating one meg, confirming the hypothesis that a unit is 2Kbyte), sbrk(0) just doesn't budge. As the MacOSX "man sbrk" says, """ The brk and sbrk functions are historical curiosities left over from earlier days before the advent of virtual memory management. """ and apparently it's now quite hard to make any USE of those quaint oddities, in presence of any attempt, anywhere in any library linked with the process, to do some "smart" memory allocation &c. > These techniques could apply to Windows with some caveats. If you are > interested in Windows, see: > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnucmg/ht > ml/UCMGch09.asp > > Can't think of anything fool-proof though. Fool-proof is way beyond what I'm looking for now -- I'd settle for "reasonably clean, works in Linux, Mac and Windows over 90% of the time, and I can detect somehow when it isn't working";-) Since people DO need to keep an eye on their code's memory consumption, I'm getting convinced that the major functional lack in today's Python standard library is some minimal set of tools to help with that task. PySizer appears to be a start in the right direction (although it may be at too early a stage to make sense for the standard library of Python 2.5), but (unless I'm missing something about it) it won't help with memory leaks not directly related to Python. Maybe we SHOULD have some function in sys to return the best guess at current memory consumption of the whole process, implemented by appropriate techniques on each platform -- right now, though, I'm trying to find out which these appropriate techniques are on today's most widespread unixoid systems, Linux and MacOSX. (As I used to be a Win32 API guru in a previous life, I'm confident that I can find out about _that_ platform by sweating enough blood on MSDN -- problem here is I don't have any Windows machine with the appropriate development system to build Python, so testing would be pretty hard, but maybe I can interest somebody who DOES have such a setup...;-) Alex -- http://mail.python.org/mailman/listinfo/python-list