[issue13959] Re-implement parts of imp in pure Python

STINNER Victor added the comment: New changeset 672b6baa71010f236ee8c8ce912e98cb542385c6 by Victor Stinner in branch 'master': bpo-32030: pass interp to _PyImport_Init() (#4736) https://github.com/python/cpython/commit/672b6baa71010f236ee8c8ce912e98cb542385c6 -- nosy: +vstinner

[issue13959] Re-implement parts of imp in pure Python

Change by STINNER Victor : -- pull_requests: +4640 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: hurray! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Since the final issue is just a nicety, I am considering this issue closed. -- dependencies: -Deprecate imp.find_module()/load_module() resolution: -> fixed ___ Python tracker ___

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset efb5e6ab10f4 by Brett Cannon in branch 'default': Issue #15167 (as part of #13959): imp.get_magic() is no implemented in http://hg.python.org/cpython/rev/efb5e6ab10f4 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Georg Brandl added the comment: Moving back to blocker for beta2. -- priority: deferred blocker -> release blocker ___ Python tracker ___ ___

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- dependencies: +Re-implement imp.get_magic() in pure Python ___ Python tracker ___ ___ Python-bugs-list ma

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- dependencies: +Implement imp.get_tag() using sys.implementation ___ Python tracker ___ ___ Python-bugs-li

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Sorry, been on vacation the past week. Georg is right, nothing left is a b1 blocker, just stuff I want to get in before 3.3 ships. -- ___ Python tracker __

[issue13959] Re-implement parts of imp in pure Python

Georg Brandl added the comment: OK, sounds like none of it would block beta1. -- priority: release blocker -> deferred blocker ___ Python tracker ___ ___

[issue13959] Re-implement parts of imp in pure Python

Changes by Eric Snow : Added file: http://bugs.python.org/file26058/issue13959_get_tag.diff ___ Python tracker ___ ___ Python-bugs-list mailin

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Here are the things I'm aware of: * implement imp.get_tag() using sys.implementation * *maybe* implement imp.get_magic() in pure Python (patch already attached here) * (low priority) find a way to rip out the bulk of PyImport_ExecCodeModuleObject() from imp.c, if p

[issue13959] Re-implement parts of imp in pure Python

Georg Brandl added the comment: Which parts are still missing here? -- nosy: +georg.brandl ___ Python tracker ___ ___ Python-bugs-lis

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 034c814eb187 by Brett Cannon in branch 'default': Issue #13959: Add to imp.find_module() and load_module's docstrings http://hg.python.org/cpython/rev/034c814eb187 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: You're right about the solution, John. I (or someone else) just needs to code it up. -- ___ Python tracker ___ _

[issue13959] Re-implement parts of imp in pure Python

John Ehresman added the comment: On 6/5/12 1:08 PM, Brett Cannon wrote: > > Brett Cannon added the comment: > > Does it work in Python 3.2, John? If it does then it's just an oversight > thanks to the lack of tests in test_imp and it shouldn't be too difficult to > support. > > But do realize

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Does it work in Python 3.2, John? If it does then it's just an oversight thanks to the lack of tests in test_imp and it shouldn't be too difficult to support. But do realize I have deprecated the function. =) -- ___

[issue13959] Re-implement parts of imp in pure Python

John Ehresman added the comment: This may be a known problem, but imp.load_module fails when trying to load a C extension. This is with the 3.3a4 release on Windows, though I suspect the problem is cross-platform. -- nosy: +jpe ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Need to update the docstrings for imp.find_module() and load_module() to mention the functions are deprecated since there is no specific raised deprecation, only documented deprecation. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 59870239813c by Brett Cannon in branch 'default': Issue #13959: Document imp.find_module/load_module as deprecated. http://hg.python.org/cpython/rev/59870239813c -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- dependencies: +Deprecate imp.find_module()/load_module() ___ Python tracker ___ ___ Python-bugs-list mail

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: I have importlib.find_loader() coded up, but getting rid of find_module() is going to be a pain thanks to pkgutil, multiprocessing.forking, modulefinder, and idlelib.EditorWindow. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 7bf8ac742d2f by Brett Cannon in branch 'default': Issue #13959: Introduce importlib.find_loader(). http://hg.python.org/cpython/rev/7bf8ac742d2f -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 626d5c6fbd95 by Brett Cannon in branch 'default': Issue #13959: Have http://hg.python.org/cpython/rev/626d5c6fbd95 -- ___ Python tracker _

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Yes. And the [:] copies the list so I don't accidentally mutate in-place (did that once already, so now I'm just being paranoid; I doubt I need it hardly anywhere I don't do +=). As for using SOURCE_SUFFIXES[0], I'm done following the ported code. I really do

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Question on this one: @@ -126,7 +131,7 @@ def load_compiled(name, pathname, file=N # XXX deprecate def load_package(name, path): if os.path.isdir(path): -extensions = _bootstrap._SOURCE_SUFFIXES + [_bootstrap._BYTECODE_SUFFIX] +extensions =

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset b81ddaf0db47 by Brett Cannon in branch 'default': Issue #13959: Deprecate imp.get_suffixes() for new attributes on http://hg.python.org/cpython/rev/b81ddaf0db47 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 22b0689346f9 by Brett Cannon in branch 'default': Issue #13959: Move module type constants to Lib/imp.py. http://hg.python.org/cpython/rev/22b0689346f9 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: OK, I'm waiting on issue #14657 (avoiding the _frozen_importlib/importlib dichotomy) is resolved before I document the new imp.extension_suffixes() as it would be good to know where I should pull in the source and bytecode suffixes. I think this only leaves get

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 257cbd2fac38 by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.get_suffixes() in Lib/imp.py. http://hg.python.org/cpython/rev/257cbd2fac38 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: That test is going to stay intermittent until issue #14657 gets resolved else the exact reason for the failure is going to be hard to debug remotely. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Benjamin Peterson added the comment: Windows is currently failing test_imp: http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/214/steps/test/logs/stdio -- nosy: +benjamin.peterson ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: So here is the deal with PyImport_ExecCodeModuleObject(): bootstrapping and Barry has made this a little annoying. =) First off, PyImport_ImportFrozenModuleObject() uses PyImport_ExecCodeModuleObject(), so that precludes just directly importing imp to handle t

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset eb68502731dd by Brett Cannon in branch 'default': Issues #13959, 14647: Re-implement imp.reload() in Lib/imp.py. http://hg.python.org/cpython/rev/eb68502731dd -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Éric Araujo added the comment: 1659 lines less than 3.2’s import.c so far! -- ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Update time! With NullImporter dealt with, that leaves get_magic(), get_tag(), reload(), and get_suffixes() as things to potentially move to Lib/imp.py. I would also like to re-implement PyImport_ExecCodeModuleObject() as it's keeping a lot of C code alive jus

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset eb5c5c23ca9b by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.NullImporter in Lib/imp.py. http://hg.python.org/cpython/rev/eb5c5c23ca9b -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: updated patch for moving TAG to importlib/imp.py -- Added file: http://bugs.python.org/file25336/issue13959_tag.diff ___ Python tracker ___

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: updated patch for magic number support in imp/importlib -- Added file: http://bugs.python.org/file25335/issue13959_magic.diff ___ Python tracker ___

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : Added file: http://bugs.python.org/file25309/explicit_path_hooks.diff ___ Python tracker ___ ___ Python-bugs-list ma

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: To try and narrow down the issue, I now have separate patches for an explicit sys.meta_path and a sys.path_hooks. Both fail on tests, but for different reasons. The explicit sys.meta_path is still failing on the __main__ issue on three of the tests and for thr

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Loosening the type restrictions is fine. As for testing, that's what the test suite is supposed to do. So if need be just write tests in Python that exercise the C code. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: How consistent do the semantics of reload() need to remain? (The C version does more type checking than the Python version probably needs to worry about. reload() seems to be one of those bits that doesn't have much test coverage.) Also, what's the best way to e

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Good feedback. The some of that code was the result of directly translating the C. I'll get a new, simpler patch up probably tomorrow night. Thanks. -- ___ Python tracker _

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: First off, you should separate the patches for get_magic() and get_tag(). Second, why is there _get_pyc_magic_int() when it is never called? Third, all of this would be greatly simplified if you just had a _RAW_MAGIC_NUMBER of 3220, did the bytes object creatio

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 4e853913054c by Brett Cannon in branch 'default': Issue #13959: Continue to try to accomodate altsep in importlib by not http://hg.python.org/cpython/rev/4e853913054c -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: After thinking about it, is MAGIC an implementation detail? It certainly reflects changes specific to the CPython interpreter. I'm much more comfortable with leaving implementation details in Python/import.c. On the other hand, there's already no small amount of

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: A patch for magic and tag. It's not quite finished, but I wanted to see if the approach was palatable. FYI, I'm also trying to push forward the sys.implementation stuff, which would help on the pyc tag. -- Added file: http://bugs.python.org/file25303/iss

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Ported _imp.reload() (Python/import.c) to Lib/imp.py. Included is the change to PyImport_ReloadModule() to make it simply a wrapper around the pure Python imp.reload(). There's a good chance I don't have this right or that I have some reference leak. I haven't

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 085cf1480cfe by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.find_module() in Lib/imp.py. http://hg.python.org/cpython/rev/085cf1480cfe -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Yeah, I'm hoping to keep pressing those odds and ends forward. I have one lingering, oddball bug in find_module, but that patch is pretty much standing on its own. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Time for a recap! I am personally not bothering with moving imp.get_tag() and imp.get_magic() to importlib._bootstrap as the amount of C code required to support the public C API is not worth it. If someone else once to do the work then by all means attach a p

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset b773a751c2e7 by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.cache_from_source() in Lib/imp.py. http://hg.python.org/cpython/rev/b773a751c2e7 New changeset ea46ebba8a0f by Brett Cannon in branch 'default': Issue #13959: Re-imple

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: I'd still like to consider this a bit more. When you're trying to understand imports, having one place to look (Lib/importlib/_bootstrap.py) is better than two, especially when the one is pure Python code. So it still may be worth it to pull in the odds and ends

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset c820aa9c0c00 by Brett Cannon in branch 'default': Issue #13959: Keep imp.get_magic() in C code, but cache in importlib http://hg.python.org/cpython/rev/c820aa9c0c00 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Looking it over, I'm confident that tokenizer.detect_encoding() does not raise a SyntaxError where PyTokenizer_FindEncodingFilename() does. I've run out of time tonight, but I'll look at it more tomorrow. Once find_module() is done, I'd like to move on to reload(

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: On the _frozen_importlib point, I'm fine with that. It was just counter-intuitive that the importlib._bootstrap functions were returning loaders whose classes weren't also defined there. I'll have another look at that test tonight and run the patch through the fu

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: You could change Lib/imp.py to have ``import _frozen_importlib as _bootstrap`` if you want the same modules coming from imp, but I would argue against changing importlib itself being changed as that complicates development since if you screw up importlib._boots

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: rather, tokenize.detect_encoding() -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: 2 "problems" with that last patch: * the types of the loaders that get returned by _bootstrap._find_module() are not the classes in _bootstrap (e.g. _frozen_importlib._SourceFileLoader). That doesn't smell right. * tokenize.get_encoding? is saying that Lib/test/b

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 66bd85bcf916 by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.load_compiled() in imp.py. http://hg.python.org/cpython/rev/66bd85bcf916 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: I doubt I will beat you to it, Eric, but I did want to say that your overall design was what I had in my head when I was thinking about how to re-implement the function, so keep at it! -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: This is a mostly-working sketch of find_module() in imp.py. I've run out of time tonight, but wanted to get it up in case it's useful to anyone else (a.k.a Brett). If nobody's touched it before then, I'll finish it up tomorrow. -- Added file: http://bugs

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 3b5b4b4bb43c by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.load_source() in imp.py. http://hg.python.org/cpython/rev/3b5b4b4bb43c -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: >From Eric Smith on python-dev: > +suffix, mode, type_ = details > +if mode and (not mode.startswith(('r', 'U'))) or '+' in mode: > +raise ValueError('invalid file open mode {!r}'.format(mode)) Should this be: if mode and (not mode.startswith(('r

[issue13959] Re-implement parts of imp in pure Python

Changes by Berker Peksag : -- nosy: +berker.peksag ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 4256df44023b by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.load_package() in imp.py. http://hg.python.org/cpython/rev/4256df44023b -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 2df37938b8e1 by Brett Cannon in branch 'default': Issue #13959: Re-implement imp.load_module() in imp.py. http://hg.python.org/cpython/rev/2df37938b8e1 -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: I am seeing how this is going to go down. the load_dynamic, load_source, etc. family of functions are simply dispatched to by load_module(). So to keep some semblance of backwards-compatibility, each of those modules need to be implemented and then have load_mo

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset 4dce3afc392c by Brett Cannon in branch 'default': Issue #13959: Simplify imp.reload() by relying on a module's http://hg.python.org/cpython/rev/4dce3afc392c -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: It looks like in order to get a clear sign of what it will take to remove various parts of import.c, imp.load_module() needs to go along with imp.load_package() (since they call each other in the C code). You also have to take care of imp.reload(), but I am sim

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: OK, so I have started to check this stuff in, but I think it's best to do it piecemeal. Going forward I would like to commit in units of functions being replaced, and prioritize stuff that cuts out C code (e.g. the load_*() methods, find_module(), etc.). That w

[issue13959] Re-implement parts of imp in pure Python

Roundup Robot added the comment: New changeset d777f854a66e by Brett Cannon in branch 'default': Issue #13959: Rename imp to _imp and add Lib/imp.py and begin http://hg.python.org/cpython/rev/d777f854a66e -- nosy: +python-dev ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Just because I was thinking about it, I wonder if necessarily all the frozen stuff really needs to stay in import.c. I mean a frozen module is really just an entry in an array of structs that has a name of an char*[]. I don't see why one couldn't simply have a

[issue13959] Re-implement parts of imp in pure Python

Antoine Pitrou added the comment: I think this should be a blocker for 3.3. -- nosy: +pitrou priority: normal -> release blocker ___ Python tracker ___ _

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: (Éric, I'd reply in the review if reitveld recognized my current username -- see http://psf.upfronthosting.co.za/roundup/meta/issue402. FYI, I also did not get an email for your review, which is likely related to that issue.) Regarding __all__, I didn't realize t

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Sounds good. It will be a while before we get there, but at that point I was already planning on getting in touch with the maintainers of relevant modules before messing with them. -- ___ Python tracker

[issue13959] Re-implement parts of imp in pure Python

Éric Araujo added the comment: > Once things all the import stuff currently on the table is wrapped up, > I'll probably work on switching things over to using importlib directly. Please leave distutils unchanged; we don’t clean it up or improve it in any way, it’s bugfixes only. Changing pack

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Of note for my patch is the addition of SUFFIXES and MODE to the 3 main "file loader" classes. I did this to reverse the dependence on imp.get_suffixes(). As well, a couple of extra functions are added to Python/importlib/_bootstrap.py for a similar reason, but s

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Here's a listing of the places where import-ish modules are used in CPython. Once things all the import stuff currently on the table is wrapped up, I'll probably work on switching things over to using importlib directly. -- Added file: http://bugs.python.

[issue13959] Re-implement parts of imp in pure Python

Eric Snow added the comment: Here's an incomplete (though essentially test-passing) patch that flips more of imp into importlib. It builds on Brett's patch. The gist is that imp.py is a minimal wrapper around importlib, which uses _imp directly. Ultimately the things implemented in Lib/impo

[issue13959] Re-implement parts of imp in pure Python

Brett Cannon added the comment: Attached is a start for re-implementing imp. Still need code for cache_from_source, source_from_cache, find_module, load_module, reload, load_compiled, load_source, and load_package. -- keywords: +patch Added file: http://bugs.python.org/file24846/imp.d

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue13959] Re-implement parts of imp in pure Python

Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue13959] Re-implement parts of imp in pure Python

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13959] Re-implement parts of imp in pure Python

Changes by Éric Araujo : -- nosy: +eric.araujo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue13959] Re-implement parts of imp in pure Python

Changes by Brett Cannon : -- dependencies: +Replace __import__ w/ importlib.__import__ ___ Python tracker ___ ___ Python-bugs-list mai

[issue13959] Re-implement parts of imp in pure Python

New submission from Brett Cannon : A bunch of code in Python/import.c exists purely for the imp module, but a large chunk of it does not need to be implemented in C but instead can be implemented in Python code. Once importlib is bootstrapped in then an attempt to scale back the C code should