[Python-Dev] Enhancement of Python memory allocators
Hi, I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory. I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue3329 Using this patch, detecting memory corruptions (buffer underflow and overflow) can be done without recompilation. We may add an environment variable to enable Python debug functions at runtime, example: PYDEBUGMALLOC=1. There is just a restriction: the environment variable would not be ignored with -E command line option, because command line options are parsed after the first memory allocation. What do you think? * The patch adds the following functions: void PyMem_GetAllocators( void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void* (**realloc_p) (void *ctx, void *ptr, size_t size), void (**free_p) (void *ctx, void *ptr)); void PyMem_SetAllocators( void *ctx, void* (*malloc) (void *ctx, size_t size), void* (*realloc) (void *ctx, void *ptr, size_t size), void (*free) (void *ctx, void *ptr)); It adds 4 similar functions (get/set) for PyObject_Malloc() and allocators of pymalloc arenas. * For the "track usage of memory" use case, see the following project which hooks memory allocators using PyMem_SetAllocators() and PyObject_SetAllocators() to get allocated bytes per filename and line number. https://pypi.python.org/pypi/pytracemalloc * Another issue proposes to use VirtualAlloc() and VirtualFree() for pymalloc arenas, see: http://bugs.python.org/issue13483 I don't know if it would be interesting, but it would now possible to choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, ...) at runtime, with an environment variable for example. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enhancement of Python memory allocators
On 13 Jun 2013 09:09, "Victor Stinner" wrote: > > Hi, > > I would like to improve memory allocators of Python. My two use cases > are replacing memory allocators with custom allocators in embedded > system and hooking allocators to track usage of memory. > > I wrote a patch for this, I'm going to commit it if nobody complains: > http://bugs.python.org/issue3329 > > Using this patch, detecting memory corruptions (buffer underflow and > overflow) can be done without recompilation. We may add an environment > variable to enable Python debug functions at runtime, example: > PYDEBUGMALLOC=1. There is just a restriction: the environment variable > would not be ignored with -E command line option, because command line > options are parsed after the first memory allocation. What do you > think? The rest of it sounds fine, but please don't add the runtime switching support to our existing main function. Interpreter startup is a mess already. If you were interested in helping directly with PEP 432, though, that would be good - I haven't been able to spend much time on it lately. Cheers, Nick. > > * > > The patch adds the following functions: > > void PyMem_GetAllocators( > void **ctx_p, > void* (**malloc_p) (void *ctx, size_t size), > void* (**realloc_p) (void *ctx, void *ptr, size_t size), > void (**free_p) (void *ctx, void *ptr)); > > void PyMem_SetAllocators( > void *ctx, > void* (*malloc) (void *ctx, size_t size), > void* (*realloc) (void *ctx, void *ptr, size_t size), > void (*free) (void *ctx, void *ptr)); > > It adds 4 similar functions (get/set) for PyObject_Malloc() and > allocators of pymalloc arenas. > > * > > For the "track usage of memory" use case, see the following project > which hooks memory allocators using PyMem_SetAllocators() and > PyObject_SetAllocators() to get allocated bytes per filename and line > number. > https://pypi.python.org/pypi/pytracemalloc > > * > > Another issue proposes to use VirtualAlloc() and VirtualFree() for > pymalloc arenas, see: > http://bugs.python.org/issue13483 > > I don't know if it would be interesting, but it would now possible to > choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, > ...) at runtime, with an environment variable for example. > > Victor > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enhancement of Python memory allocators
2013/6/13 Nick Coghlan : > On 13 Jun 2013 09:09, "Victor Stinner" wrote: >> Using this patch, detecting memory corruptions (buffer underflow and >> overflow) can be done without recompilation. We may add an environment >> variable to enable Python debug functions at runtime, example: >> PYDEBUGMALLOC=1. There is just a restriction: the environment variable >> would not be ignored with -E command line option, because command line >> options are parsed after the first memory allocation. What do you >> think? > > The rest of it sounds fine, but please don't add the runtime switching > support to our existing main function. Interpreter startup is a mess > already. If you were interested in helping directly with PEP 432, though, > that would be good - I haven't been able to spend much time on it lately. I proposed an environment variable to solve the following issue: when memory allocators are replaced with custom allocators, debug hooks cannot be used. Debug hooks must be set before the first memory allocation. Another option is to add a new function (ex: PyMem_SetDebugHook()) to install explicitly debug hooks, so it can be called after PyMem_SetAllocators() and before the first memory allocation. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Move test___all__ over to unittest.main() and use ModuleNotFoundError
Obviously the commit message is a little misleading since changes I was
about to stage accidentally went in on this change. Sorry about that. Same
basic concept of the changes, just to more modules.
On Wed, Jun 12, 2013 at 8:12 PM, brett.cannon wrote:
> http://hg.python.org/cpython/rev/c4d7228421df
> changeset: 84106:c4d7228421df
> user:Brett Cannon
> date:Wed Jun 12 20:12:30 2013 -0400
> summary:
> Move test___all__ over to unittest.main() and use ModuleNotFoundError
>
> files:
> Lib/test/regrtest.py | 16
> Lib/test/support.py | 14 +++---
> Lib/test/test___all__.py | 7 ++-
> Lib/xmlrpc/server.py | 2 +-
> Lib/zipfile.py | 6 +++---
> 5 files changed, 21 insertions(+), 24 deletions(-)
>
>
> diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
> --- a/Lib/test/regrtest.py
> +++ b/Lib/test/regrtest.py
> @@ -146,11 +146,11 @@
>
> try:
> import threading
> -except ImportError:
> +except ModuleNotFoundError:
> threading = None
> try:
> import multiprocessing.process
> -except ImportError:
> +except ModuleNotFoundError:
> multiprocessing = None
>
>
> @@ -180,7 +180,7 @@
> if sys.platform == 'darwin':
> try:
> import resource
> -except ImportError:
> +except ModuleNotFoundError:
> pass
> else:
> soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
> @@ -571,7 +571,7 @@
> if findleaks:
> try:
> import gc
> -except ImportError:
> +except ModuleNotFoundError:
> print('No GC available, disabling findleaks.')
> findleaks = False
> else:
> @@ -692,7 +692,7 @@
> if use_mp:
> try:
> from threading import Thread
> -except ImportError:
> +except ModuleNotFoundError:
> print("Multiprocess option requires thread support")
> sys.exit(2)
> from queue import Queue
> @@ -1396,7 +1396,7 @@
> pic = sys.path_importer_cache.copy()
> try:
> import zipimport
> -except ImportError:
> +except ModuleNotFoundError:
> zdc = None # Run unmodified on platforms without zipimport support
> else:
> zdc = zipimport._zip_directory_cache.copy()
> @@ -1473,7 +1473,7 @@
> sys.path_importer_cache.update(pic)
> try:
> import zipimport
> -except ImportError:
> +except ModuleNotFoundError:
> pass # Run unmodified on platforms without zipimport support
> else:
> zipimport._zip_directory_cache.clear()
> @@ -1510,7 +1510,7 @@
> doctest.master = None
> try:
> import ctypes
> -except ImportError:
> +except ModuleNotFoundError:
> # Don't worry about resetting the cache if ctypes is not supported
> pass
> else:
> diff --git a/Lib/test/support.py b/Lib/test/support.py
> --- a/Lib/test/support.py
> +++ b/Lib/test/support.py
> @@ -29,27 +29,27 @@
>
> try:
> import _thread, threading
> -except ImportError:
> +except ModuleNotFoundError:
> _thread = None
> threading = None
> try:
> import multiprocessing.process
> -except ImportError:
> +except ModuleNotFoundError:
> multiprocessing = None
>
> try:
> import zlib
> -except ImportError:
> +except ModuleNotFoundError:
> zlib = None
>
> try:
> import bz2
> -except ImportError:
> +except ModuleNotFoundError:
> bz2 = None
>
> try:
> import lzma
> -except ImportError:
> +except ModuleNotFoundError:
> lzma = None
>
> __all__ = [
> @@ -116,7 +116,7 @@
> with _ignore_deprecated_imports(deprecated):
> try:
> return importlib.import_module(name)
> -except ImportError as msg:
> +except ModuleNotFoundError as msg:
> if sys.platform.startswith(tuple(required_on)):
> raise
> raise unittest.SkipTest(str(msg))
> @@ -188,7 +188,7 @@
> if not _save_and_block_module(blocked_name, orig_modules):
> names_to_remove.append(blocked_name)
> fresh_module = importlib.import_module(name)
> -except ImportError:
> +except ModuleNotFoundError:
> fresh_module = None
> finally:
> for orig_name, module in orig_modules.items():
> diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py
> --- a/Lib/test/test___all__.py
> +++ b/Lib/test/test___all__.py
> @@ -75,7 +75,7 @@
> try:
> import rlcompleter
> import locale
> -except ImportError:
> +except ModuleNotFoundError:
> pass
> else:
> locale.setlocale(locale.LC_CTYPE, 'C')
> @@ -113,8 +113,5 @@
> print('Following modules failed to be imported:',
> failed_imports)
>
>
> -def test_main():
> -support.run_unittest(AllTest)
> -
> if __name__ == "__main__":
> -test_main()
> +unittes
Re: [Python-Dev] Enhancement of Python memory allocators
On 13 Jun 2013 10:09, "Victor Stinner" wrote: > > 2013/6/13 Nick Coghlan : > > On 13 Jun 2013 09:09, "Victor Stinner" wrote: > >> Using this patch, detecting memory corruptions (buffer underflow and > >> overflow) can be done without recompilation. We may add an environment > >> variable to enable Python debug functions at runtime, example: > >> PYDEBUGMALLOC=1. There is just a restriction: the environment variable > >> would not be ignored with -E command line option, because command line > >> options are parsed after the first memory allocation. What do you > >> think? > > > > The rest of it sounds fine, but please don't add the runtime switching > > support to our existing main function. Interpreter startup is a mess > > already. If you were interested in helping directly with PEP 432, though, > > that would be good - I haven't been able to spend much time on it lately. > > I proposed an environment variable to solve the following issue: when > memory allocators are replaced with custom allocators, debug hooks > cannot be used. Debug hooks must be set before the first memory > allocation. > > Another option is to add a new function (ex: PyMem_SetDebugHook()) to > install explicitly debug hooks, so it can be called after > PyMem_SetAllocators() and before the first memory allocation. Yes, that sounds better. One of the biggest problems with the current startup sequence is the way it relies on environment variables for configuration, which makes life hard for other applications that want to embed the CPython runtime. Cheers, Nick. > > Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
