[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: By the way, you may be interested to learn that the patch in issue10241 has made things quite a bit better now: C extension modules can be collected much earlier. -- ___ Python tracker

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Ok, let's attack the rest separately then :) And thanks a lot for testing! -- status: open -> closed ___ Python tracker ___

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: Yes, I agree the patch is ok. It would be would be much simpler to keep track of the module dicts if they were weakrefable. Alternatively, at shutdown a weakrefable object with a reference to the module dict could be inserted in to each module dict. We cou

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: > Also, do note that purge/gc after wiping can still be a regular gc > pass unless the module has been wiped. The gc could be triggered by > another module being wiped. That said, I welcome any suggestions to improve things. The ultimate reasons we need to purge

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: > Also, do note that purge/gc after wiping can still be a regular > gc pass unless the module has been wiped. The gc could be triggered > by another module being wiped. For me, the modules which die naturally after purging begins are # purge/gc encodings.alias

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Here (Linux) I get the following: # purge/gc os.path 12 # purge/gc operator 50 # purge/gc io 49 # purge/gc _sysconfigdata 48 # purge/gc sysconfig 47 # purge/gc keyword 46 # purge/gc site 45 # purge/gc types 44 Also, do note that purge/gc after wiping can still

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: > I get different numbers from you. If I run "./python -v -c pass", most > modules in the "wiping" phase are C extension modules, which is expected. > Pretty much every pure Python module ends up garbage collected before > that. The *module* gets gc'ed, sure

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: > You might want to open a prompt and look at gc.get_referrers() for > encodings.mbcs.__dict__ (or another of those modules). >>> gc.get_referrers(sys.modules['encodings.mbcs'].__dict__) [, , , ] >>> gc.get_referrers(sys.modules['encodings.cp1252'].__dict__)

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: You might want to open a prompt and look at gc.get_referrers() for encodings.mbcs.__dict__ (or another of those modules). -- ___ Python tracker ___

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: On 01/08/2013 10:59am, Antoine Pitrou wrote: > If you replace the end of your script with the following: > > for name, mod in sys.modules.items(): > if name != 'encodings': > mod.__dict__["__blob__"] = Blob(name) > del name, mod, Blob > > > then at

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: > As for the __main__ dict, perhaps we're missing a decref somewhere. Actually, it's not surprising. Blob's methods hold a reference to the __main__ globals, and there's still a Blob object alive in encodings. If you replace the end of your script with the fol

[issue18214] Stop purging modules which are garbage collected before shutdown

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: > It appears that a few (1-3) module dicts are not being purged because they > have been "orphaned". (i.e. the module object was garbaged collected before > we check the weakref, but the module dict survived.) Module globals can be kept alive by any function

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: I played a bit with the patch and -v -Xshowrefcount. The number of references and blocks left at exit varies (and is higher than for unpatched python). It appears that a few (1-3) module dicts are not being purged because they have been "orphaned". (i.e. th

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Let's wait for the buildbots on this one too. -- resolution: -> fixed stage: patch review -> committed/rejected ___ Python tracker ___

[issue18214] Stop purging modules which are garbage collected before shutdown

Roundup Robot added the comment: New changeset 79e2f5bbc30c by Antoine Pitrou in branch 'default': Issue #18214: Improve finalization of Python modules to avoid setting their globals to None, in most cases. http://hg.python.org/cpython/rev/79e2f5bbc30c -- nosy: +python-dev

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Updated patch with a hack in Lib/site to unpatch builtins early at shutdown. -- Added file: http://bugs.python.org/file31101/module_cleanup5.patch ___ Python tracker __

[issue18214] Stop purging modules which are garbage collected before shutdown

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Updated patch has tests and also removes several cleanup hacks. -- Added file: http://bugs.python.org/file31099/module_cleanup4.patch ___ Python tracker ___

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: See issue10068 and issue7140. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue18214] Stop purging modules which are garbage collected before shutdown

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file31093/module_cleanup3.patch ___ Python tracker ___ ___ Python-bugs-list mai

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Slightly better patch. Also, as I pointed out in python-dev (http://mail.python.org/pipermail/python-dev/2013-July/127673.html), this is still imperfect due to various ways modules can be kept alive from long-lived C variables. -- Added file: http://

[issue18214] Stop purging modules which are garbage collected before shutdown

Changes by Antoine Pitrou : -- components: +Interpreter Core nosy: +amaury.forgeotdarc, gregory.p.smith stage: -> patch review type: -> enhancement ___ Python tracker ___ _

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: Now that PEP 442 is committed, here is the patch. -- Added file: http://bugs.python.org/file31090/module_cleanup.patch ___ Python tracker ___ __

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: > > As for the interpreter shutdown itself, I have a pending patch (post-PEP > > 442) > > to get rid of the globals cleanup as well. It may be better to merge the > > two approaches. > > So you would just depend on garbage collection? No, I also clean up thos

[issue18214] Stop purging modules which are garbage collected before shutdown

Richard Oudkerk added the comment: On 15/06/2013 7:11pm, Antoine Pitrou wrote: >> Usually garbage collection will end up clearing the module's dict anyway. > > This is not true, since global objects might have a __del__ and then hold > the whole module dict alive through a reference cycle. Happil

[issue18214] Stop purging modules which are garbage collected before shutdown

Antoine Pitrou added the comment: > Usually garbage collection will end up clearing the module's dict anyway. This is not true, since global objects might have a __del__ and then hold the whole module dict alive through a reference cycle. Happily though, PEP 442 is going to make that concern o

[issue18214] Stop purging modules which are garbage collected before shutdown

Changes by Richard Oudkerk : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue18214] Stop purging modules which are garbage collected before shutdown

New submission from Richard Oudkerk: Currently when a module is garbage collected its dict is purged by replacing all values except __builtins__ by None. This helps clear things at shutdown. But this can cause problems if it occurs *before* shutdown: if we use a function defined in a module