Sébastien Sablé <sa...@users.sourceforge.net> added the comment: > I'm just proposing an alternative that I find cleaner, simpler and easier to > maintain.
I understand how LD_PRELOAD works but I find it neither clean nor simple to maintain. Also by using a wrapper to call Python you still impact all the applications that may be executed from Python since the environment variables are propagated. You also need to configure the path to the alternative malloc library at runtime. And as I said above, I am afraid most AIX and SunOS users will never hear about that and will just use the default behavior, with their Python application taking much more memory than necessary. As mentioned by Floris, AIX is being pushed by IBM quite a lot, and in some markets it is very common (if not predominant in finance for example - 50/50 with SunOS for my clients I would say). > I mean, if you start embedding malloc in python, why stop there, and not embed the whole glibc ;-) Concerning AIX, that would not be such a bad idea given the number of bugs in the native C library (cf some of my other issues reported in python bug tracker) - just kidding ;-) Concerning the fact that dlmalloc or ptmalloc evolve "quickly": * dlmalloc V2.8.3 Thu Sep 22 11:16:32 2005 * ptmalloc2 release Jun 5th, 2006 * ptmalloc3 release May 31st, 2006 * dlmalloc V2.8.4 Wed May 27 09:56:23 2009 I think we can cope with that kind of "fast" evolution ;-) Also an old dlmalloc is better than no dlmalloc at all. And as you noticed, an old dlmalloc is already provided in libffi. > So how about a --with-dlmalloc=path/to/dlmalloc.c? That looks like a good alternative. I can implement that if that can help to get the patch in Python. > Can't you just add dlmalloc to LDFLAGS or something? Or would the default malloc still be selected? There is a USE_DL_PREFIX in malloc.c. If this flag is defined, all functions will be prefixed by dl (dlmalloc, dlfree, dlrealloc...). If it is not set, the functions will be named as usual (malloc, free...). In my patch, I preferred to set USE_DL_PREFIX and call dlmalloc/dlfree explicitly where needed. Since I want PyMem_MALLOC to call dlmalloc, I would need to export the "malloc" symbol from libpython so that Python extensions could use it when calling PyMem_MALLOC, but that would impact all malloc calls in applications which embed Python for example. So I think it is probably better to explicitly distinguish when you want to call dlmalloc and leave the native malloc for the host application. Also this only addresses the --with-dlmalloc part of my patch. The other part concerning --with-pymalloc-mmap ensures that pymalloc uses mmap to allocate arenas rather than malloc. I perfectly understand that people are reluctant to make the memory allocation system more complex than it is already in Python in order to bypass some limitations of systems which are not very widespread among Python users. But Python eating a lot of memory on SunOS and AIX does not look very good either. I have some strong requirements as far as memory is concerned for my application so I have maintained this patch internally and distributed it as part of my application. I will probably change of job soon and will not have access to AIX systems anymore. I don't really expect this patch to be accepted soon as few people have expressed some interest and I don't have much time/interest to push it on python-dev, but I will update the patch for Python 2.7 and 3.2 before leaving so that people impacted by this problem could a least manually patch their Python if they find this issue. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue3526> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
