Re: [Python-Dev] Zipping the standard library.
On Sat, Mar 10, 2012 at 22:16, PJ Eby wrote: > On Sat, Mar 10, 2012 at 5:49 PM, Thomas Wouters wrote: > >> (And, yes, I'm zipping up the stdlib for Python 2.7 at Google, to reduce >> the impact on the aforementioned million of machines :) >> > > You might want to consider instead backporting the importlib caching > facility, since it provides some of the zipimport benefits for plain old, > non-zipped modules. Actually, a caching-only import hook that operated > that way wouldn't even need the whole of importlib, just a wrapper over the > standard C import that skips the unnecessary filesystem accesses. > Thanks for the suggestions (Antoine too), but that's not really the topic I want to discuss here (but if you guys move to Google I'll happily discuss all the stuff we have to deal with.) The questions is really whether Python wants to actually support zipped stdlibs or not. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] im_func: implementation detail?
How does someone know if something in CPython is an implementation detail or not? In the case of im_func, I think it is (an implementation detail), and another person thinks it is part of the language spec. That all implementations must have. ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] im_func: implementation detail?
On Sun, Mar 11, 2012 at 1:11 PM, Ethan Furman wrote: > How does someone know if something in CPython is an implementation detail or > not? Sadly, there's a large grey area where the language reference doesn't have enough rigor, so asking here is often the only way. > In the case of im_func, I think it is (an implementation detail), and > another person thinks it is part of the language spec. That all > implementations must have. It's part of the language spec. However it's now called __func__. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Zipping the standard library.
On Sun, Mar 11, 2012 at 12:26 PM, Thomas Wouters wrote: > Thanks for the suggestions (Antoine too), but that's not really the topic I > want to discuss here (but if you guys move to Google I'll happily discuss > all the stuff we have to deal with.) The questions is really whether Python > wants to actually support zipped stdlibs or not. I do want to support it; that's why we put the facilities you found there in the first place. Unfortunately nobody actually did the necessary second step of trying to bundle the stdlib and trying to make the tests pass. So I think it would be great if we addressed the issues you found, or at least started prioritizing them. I'm not sure if you're saying that you're hitting the 2 GB limit *with just the stdlib* in a zipfile, or if you're hitting this after you've added a bunch of Google code to it as well. I'm also not sure that it's worth the effort to make *all* the tests in the stdlib pass -- some tests may just be testing filesystem things that make no sense when the stdlib is in a zipfile. I see you frowning already about my lax attitide... So let me add that all non-test code should definitely work, and quite possible the only way to ensure that this is the case is to make all the tests pass. The issue with needing os.py outside the zipfile is a good thing to try to fix. The importlib and zipfile issues don't worry me particularly, but depending on your answer about the 2 GB limit I might get more concerned. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Zipping the standard library.
On Mon, Mar 12, 2012 at 7:08 AM, Guido van Rossum wrote: > I do want to support it; that's why we put the facilities you found > there in the first place. Unfortunately nobody actually did the > necessary second step of trying to bundle the stdlib and trying to > make the tests pass. So I think it would be great if we addressed the > issues you found, or at least started prioritizing them. This is the main stdlib API designed to avoid the need to make the filesystem imports assumption (as you can also use it to read source files): http://docs.python.org/py3k/library/pkgutil#pkgutil.get_data Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Zipping the standard library.
On Sun, Mar 11, 2012 at 14:08, Guido van Rossum wrote: > On Sun, Mar 11, 2012 at 12:26 PM, Thomas Wouters > wrote: > > Thanks for the suggestions (Antoine too), but that's not really the > topic I > > want to discuss here (but if you guys move to Google I'll happily discuss > > all the stuff we have to deal with.) The questions is really whether > Python > > wants to actually support zipped stdlibs or not. > > I do want to support it; that's why we put the facilities you found > there in the first place. Unfortunately nobody actually did the > necessary second step of trying to bundle the stdlib and trying to > make the tests pass. So I think it would be great if we addressed the > issues you found, or at least started prioritizing them. > > I'm not sure if you're saying that you're hitting the 2 GB limit *with > just the stdlib* in a zipfile, or if you're hitting this after you've > added a bunch of Google code to it as well. No, not with just the stdlib, but in a Google binary that embeds Python -- the 32-bit-unsigned numbers in zipfiles are file offsets, so in a Google binary (which, as you know, is typically a completely statically linked binary) the offsets for a zipfile embedded in the binary can be bigger than that. (If you were thinking of PAR files, those don't use zipimport themselves, but their own PEP-302 importer written in Python with the zipfile module, so it's okay.) > I'm also not sure that > it's worth the effort to make *all* the tests in the stdlib pass -- > some tests may just be testing filesystem things that make no sense > when the stdlib is in a zipfile. I see you frowning already about my > lax attitide... Hah, no, I wasn't frowning when I read that :) I don't care about making all tests pass, but I do want them to not fail -- a test should only fail if the tested thing doesn't work, not if the test can't run. For what it's worth, the vast majority of tests work fine, there's just a couple that take what I would call unwarranted assumptions. For example, the zipfile module wants to test the writepy method, so it needs a module and a package to bundle in the zipfile. It could make a bunch of tempfiles (as most other tests do) into a package, but instead it uses email.__file__ to find the email package and uses that. The only failing test I remember that wasn't of the kind of using the stdlib source out of laziness is test_pyclbr, which runs pyclbr over a whole bunch of large stdlib modules. It also does other tests, so I don't think skipping the test for a zipped stdlib is a big deal, but even that could be fixed by using PEP 302's interface for getting the source. Of course, we also have to consider that the zipped stdlib may contain just .pyc files :) So it's definitely possible to fix most tests, possibly all of them, without too much effort. > So let me add that all non-test code should definitely > work, and quite possible the only way to ensure that this is the case > is to make all the tests pass. The issue with needing os.py outside > the zipfile is a good thing to try to fix. > I forgot to include a link to http://bugs.python.org/issue12919 that makes that a little less confusing (to me, although others apparently disagreed :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Where to discuss PEP 382 vs. PEP 402 (namespace packages)?
Martin has asked me to decide on PEP 382 vs. PEP 402 (namespace packages) in time for inclusion of the decision in Python 3.3. As people who attended the language-sig know, I am leaning towards PEP 402 but I admit that at this point I don't have enough information. If I have questions, should I be asking them on the import-sig or on python-dev? Is it tolerable if I ask questions even if the answer is somewhere in the archives? (I spent a lot of time reviewing the "pitchfork thread", http://mail.python.org/pipermail/python-dev/2006-April/064400.html, but that wasn't particularly fruitful, so I'm worried I'd just waste my time browsing the archives -- if the PEP authors did their jobs well the PEPs should include summaries of the discussion anyways.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Import-SIG] Where to discuss PEP 382 vs. PEP 402 (namespace packages)?
I think restarting the discussion anew here on distutils-sig is appropriate. -- Eric. Guido van Rossum wrote: Martin has asked me to decide on PEP 382 vs. PEP 402 (namespace packages) in time for inclusion of the decision in Python 3.3. As people who attended the language-sig know, I am leaning towards PEP 402 but I admit that at this point I don't have enough information. If I have questions, should I be asking them on the import-sig or on python-dev? Is it tolerable if I ask questions even if the answer is somewhere in the archives? (I spent a lot of time reviewing the "pitchfork thread", http://mail.python.org/pipermail/python-dev/2006-April/064400.html, but that wasn't particularly fruitful, so I'm worried I'd just waste my time browsing the archives -- if the PEP authors did their jobs well the PEPs should include summaries of the discussion anyways.) -- --Guido van Rossum (python.org/~guido) _ Import-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/import-sig ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Import-SIG] Where to discuss PEP 382 vs. PEP 402 (namespace packages)?
And of course I meant import-sig. -- Eric. "Eric V. Smith" wrote: I think restarting the discussion anew here on distutils-sig is appropriate. -- Eric. Guido van Rossum wrote: Martin has asked me to decide on PEP 382 vs. PEP 402 (namespace packages) in time for inclusion of the decision in Python 3.3. As people who attended the language-sig know, I am leaning towards PEP 402 but I admit that at this point I don't have enough information. If I have questions, should I be asking them on the import-sig or on python-dev? Is it tolerable if I ask questions even if the answer is somewhere in the archives? (I spent a lot of time reviewing the "pitchfork thread", http://mail.python.org/pipermail/python-dev/2006-April/064400.html, but that wasn't particularly fruitful, so I'm worried I'd just waste my time browsing the archives -- if the PEP authors did their jobs well the PEPs should include summaries of the discussion anyways.) -- --Guido van Rossum (python.org/~guido) _ Import-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/import-sig ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Where to discuss PEP 382 vs. PEP 402 (namespace packages)?
Am 11.03.12 16:02, schrieb Guido van Rossum: Martin has asked me to decide on PEP 382 vs. PEP 402 (namespace packages) in time for inclusion of the decision in Python 3.3. As people who attended the language-sig know, I am leaning towards PEP 402 but I admit that at this point I don't have enough information. If I have questions, should I be asking them on the import-sig or on python-dev? import-sig would be best. Is it tolerable if I ask questions even if the answer is somewhere in the archives? Sure! Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
