Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
> The main question is if anyone ever used Py_TRACE_REFS? Does someone > use sys.getobjects() or PYTHONDUMPREFS environment variable? I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy. We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562. I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful. -sully ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 591 discussion (final qualifier) happening at typing-sig@
I've submitted PEP 591 (Adding a final qualifier to typing) for discussion to typing-sig [1]. Here's the abstract: This PEP proposes a "final" qualifier to be added to the ``typing`` module---in the form of a ``final`` decorator and a ``Final`` type annotation---to serve three related purposes: * Declaring that a method should not be overridden * Declaring that a class should not be subclassed * Declaring that a variable or attribute should not be reassigned Full text at https://www.python.org/dev/peps/pep-0591/ -sully [1] https://mail.python.org/mailman3/lists/typing-sig.python.org/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith wrote: > On Mon, Apr 15, 2019, 15:27 Michael Sullivan wrote: > >> > The main question is if anyone ever used Py_TRACE_REFS? Does someone >> > use sys.getobjects() or PYTHONDUMPREFS environment variable? >> >> I used sys.getobjects() today to track down a memory leak in the >> mypyc-compiled version of mypy. >> >> We were leaking memory badly but no sign of the leak was showing up in >> mypy's gc.get_objects() based profiler. Using a debug build and switching >> to sys.getobjects() showed that we were badly leaking int objects. A quick >> inspection of the values in question (large and random looking) suggested >> we were leaking hash values, and that quickly pointed me to >> https://github.com/mypyc/mypyc/pull/562. >> >> I don't have any strong feelings about whether to keep it in the >> "default" debug build, though. I was using a debug build that I built >> myself with every debug feature that seemed potentially useful. >> > > This is mostly to satisfy my curiosity, so feel free to ignore: did you > try using address sanitizer or valgrind? > > I didn't, mostly because I assume that valgrind wouldn't play well with cpython. (I've never used address sanitizer.) I was curious, so I went back and tried it out. It turned out to not seem to need that much fiddling to get to work. It slows things down a *lot* and produced 17,000 "loss records", though, so maybe I don't have it working right. At a glance the records did not shed any light. I'd definitely believe that valgrind is up to the task of debugging this, but my initial take with it shed much less light than my sys.getobjects() approach. (Though note that my sys.getobjects() approach was slotting it into an existing python memory profiler we had hacked up, so...) -sully > -n > >> ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 591 discussion (final qualifier) happening at typing-sig@
On Mon, Apr 15, 2019 at 8:12 PM Nathaniel Smith wrote: > On Mon, Apr 15, 2019 at 5:00 PM Michael Sullivan wrote: > > > > I've submitted PEP 591 (Adding a final qualifier to typing) for > discussion to typing-sig [1]. > > I'm not on typing-sig [1] so I'm replying here. > > > Here's the abstract: > > This PEP proposes a "final" qualifier to be added to the ``typing`` > > module---in the form of a ``final`` decorator and a ``Final`` type > > annotation---to serve three related purposes: > > > > * Declaring that a method should not be overridden > > * Declaring that a class should not be subclassed > > * Declaring that a variable or attribute should not be reassigned > > I've been meaning to start blocking subclassing at runtime (e.g. like > [2]), so being able to express that to the typechecker seems like a > nice addition. I'm assuming though that the '@final' decorator doesn't > have any runtime effect, so I'd have to say it twice? > > @typing.final > class MyClass(metaclass=othermod.Final): > ... > > Or on 3.6+ with __init_subclass__, it's easy to define a @final > decorator that works at runtime, but I guess this would have to be a > different decorator? > > @typing.final > @alsoruntime.final > class MyClass: > ... > > This seems kinda awkward. Have you considered giving it a runtime > effect, or providing some way for users to combine these two things > together on their own? > > Nothing else in typing does any type of runtime enforcement, so I'd be reluctant to start here. One approach would be doing something like this (maybe in a support module): if typing.TYPE_CHECKING: from typing import final else: from alsoruntime import final So that at checking time, the typechecker would use the typing final but at runtime we'd get something that does enforcement. (And for the pre-3.6 case, you could maybe use something like six.add_metaclass in order to specify the metaclass as a decorator.) I can add this as an example to the PEP. -sully > -n > > [1] https://github.com/willingc/pep-communication/issues/1 > [2] https://stackoverflow.com/a/3949004/1925449 > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
On Tue, Apr 16, 2019 at 2:11 AM Victor Stinner wrote: > Hi Michael, > > Do you know the tracemalloc module? Did you try it? It works on a > regular Python (compiled in debug mode). > > I would be curious to know if tracemalloc also allows you to track the > memory leak. > > Playing around with it a little it does not seem super helpful here (unless I am missing something): it tracks the allocations based on the python call stack, which doesn't help here, in a C extension module generated from python code. Though, in the the mypyc case, we could implement a debug option for creating dummy frames so that we always have a useful call stack. That seems like less of an option for actual hand-written extension modules, though. (Though on the flip side, the python call stacks might be more useful there.) sys.getobjects() is just a list of objects. Do you have a tool written > on top of it to track memory leaks? If yes, how? > > Not really. We have a very simple memory profiler built on top of gc.get_objects() that just reports how many of different types of objects there are and how much memory they are using: https://github.com/python/mypy/blob/master/mypy/memprofile.py. I swapped out gc.get_objects() for sys.getobjects(), observed that we were leaking int objects, and inspected the live int objects, which gave a pretty good clue where the leak was. > Victor > > Le mar. 16 avr. 2019 à 00:28, Michael Sullivan a écrit > : > > > > > The main question is if anyone ever used Py_TRACE_REFS? Does someone > > > use sys.getobjects() or PYTHONDUMPREFS environment variable? > > > > I used sys.getobjects() today to track down a memory leak in the > mypyc-compiled version of mypy. > > > > We were leaking memory badly but no sign of the leak was showing up in > mypy's gc.get_objects() based profiler. Using a debug build and switching > to sys.getobjects() showed that we were badly leaking int objects. A quick > inspection of the values in question (large and random looking) suggested > we were leaking hash values, and that quickly pointed me to > https://github.com/mypyc/mypyc/pull/562. > > > > I don't have any strong feelings about whether to keep it in the > "default" debug build, though. I was using a debug build that I built > myself with every debug feature that seemed potentially useful. > > > > -sully > > ___ > > Python-Dev mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
