[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Committed now (with invalidate_caches() as well). -- resolution: -> fixed stage: commit review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue14043] Speed-up importlib's _FileFinder

Roundup Robot added the comment: New changeset bbaab666e6c7 by Antoine Pitrou in branch 'default': Issue #14043: Speed up importlib's _FileFinder by at least 8x, and add a new importlib.invalidate_caches() function. http://hg.python.org/cpython/rev/bbaab666e6c7 -- nosy: +python-dev __

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: I wouldn't worry too much about test_runpy as it has always been a touchy test for me. -- ___ Python tracker ___ ___

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: This is a patch demonstrating invalidate_caches(). As for test_runpy, it seems runpy uses imp.get_loader(), so I don't see how it can remain synchronized with importlib's state. -- Added file: http://bugs.python.org/file24560/listdir_cache3.patch

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Another possibility would be to include an explicit invalidate_caches() function in importlib. -- ___ Python tracker ___ _

[issue14043] Speed-up importlib's _FileFinder

Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > Well, it's the reason you added file size to .pyc files. =) pyc files only store the integer timestamp. Here I store the original floating-point timestamp, and I assumed it would be sufficient :) > I assume if you store the system clock as well to see if no

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: Well, it's the reason you added file size to .pyc files. =) I assume if you store the system clock as well to see if no time has changed it would kill the performance gain. We could also have two versions of _FileFinder such that people could choose which they

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > The lastest patch (listdir_cache2) LGTM. Have you by any chance > patched it in and run ./python -m importlib.test.regrtest (runs the > entire test suite with builtins.__import__() substituted with > importlib.__import__())? There are a couple of test failure

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: The lastest patch (listdir_cache2) LGTM. Have you by any chance patched it in and run ./python -m importlib.test.regrtest (runs the entire test suite with builtins.__import__() substituted with importlib.__import__())? If not I can do it on my ride home tonight

[issue14043] Speed-up importlib's _FileFinder

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file24552/listdir_cache2.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Updated light-weight caching patch (now checks PYTHONCASEOK dynamically). -- ___ Python tracker ___ ___

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > And while I'm thinking about it, the hybrid caching approach of > caching just the directory contents to start and then caching full > details on successful searches (and failures) would also let you cache > the full file path as the strings will be interned a

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: And while I'm thinking about it, the hybrid caching approach of caching just the directory contents to start and then caching full details on successful searches (and failures) would also let you cache the full file path as the strings will be interned and used

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: On Fri, Feb 17, 2012 at 14:31, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > > Why pre-calculate everything? In the most common case any single > > module will be imported once, if at all. And once it is imported it > > will get cached in sys.m

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Patch with light-weight alternative. -- Added file: http://bugs.python.org/file24551/listdir_cache.patch ___ Python tracker ___ ___

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > Why pre-calculate everything? In the most common case any single > module will be imported once, if at all. And once it is imported it > will get cached in sys.modules, alleviating the need to hit the finder > again. So from a performance standpoint wouldn't i

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: OK, I have gone ahead and done a review over on rietveld for the code as-is. But I do have a design question. Why pre-calculate everything? In the most common case any single module will be imported once, if at all. And once it is imported it will get cached in

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > Just to fill you in, the discussion centred on the idea of doing a > listdir() of the directory the FileFinder was in charge of watching > and caching that. Then, when it had to look up a file all it had to do > was stat the directory to look for a change befo

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: Just to fill you in, the discussion centred on the idea of doing a listdir() of the directory the FileFinder was in charge of watching and caching that. Then, when it had to look up a file all it had to do was stat the directory to look for a change before it s

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Regenerated patch. -- Added file: http://bugs.python.org/file24550/find_module_cache3.patch ___ Python tracker ___

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: > I have not had a chance to deep-dive in the patch, but I just wanted > to double-check that this (a) implemented the idea PJE and I discussed > on python-dev I haven't followed this discussion. > , and (b) you re-generate the patch as I pushed some changes t

[issue14043] Speed-up importlib's _FileFinder

Brett Cannon added the comment: I have not had a chance to deep-dive in the patch, but I just wanted to double-check that this (a) implemented the idea PJE and I discussed on python-dev, and (b) you re-generate the patch as I pushed some changes to importlib today. -- __

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Updated patch with faster cache filling. -- Added file: http://bugs.python.org/file24549/find_module_cache2.patch ___ Python tracker ___ __

[issue14043] Speed-up importlib's _FileFinder

Antoine Pitrou added the comment: Note that the cost of filling the cache itself isn't trivial: $ ./python -m timeit -s "import sys; from importlib import _bootstrap" "_bootstrap._file_path_hook('Lib')._fill_cache()" 1000 loops, best of 3: 1.88 msec per loop -- __

[issue14043] Speed-up importlib's _FileFinder

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file24548/find_module_cache.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue14043] Speed-up importlib's _FileFinder

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file24547/find_module_cache.patch ___ Python tracker ___ ___ Python-bugs-list

[issue14043] Speed-up importlib's _FileFinder

New submission from Antoine Pitrou : This patch makes importlib's _FileFinder more than 10x faster by keeping a cache of directory entries. It's actually faster than imp.find_module()! -> imp.find_module: $ ./python -m timeit -s "import imp" "imp.find_module('logging', None)" 1 loops, best