Re: [Python-Dev] PEP 380
On 26/11/2011 07:46, Maciej Fijalkowski wrote: On Sat, Nov 26, 2011 at 6:39 AM, Nick Coghlan wrote: On Sat, Nov 26, 2011 at 8:14 AM, Michael Foord wrote: On 24 Nov 2011, at 04:06, Nick Coghlan wrote: On Thu, Nov 24, 2011 at 10:28 AM, Guido van Rossum wrote: Mea culpa for not keeping track, but what's the status of PEP 380? I really want this in Python 3.3! There are two relevant tracker issues (both with me for the moment). The main tracker issue for PEP 380 is here: http://bugs.python.org/issue11682 That's really just missing the doc updates - I haven't had a chance to look at Zbyszek's latest offering on that front, but it shouldn't be far off being complete (the *text* in his previous docs patch actually seemed reasonable - I mainly objected to way it was organised). However, the PEP 380 test suite updates have a dependency on a new dis module feature that provides an iterator over a structured description of bytecode instructions: http://bugs.python.org/issue11816 Is it necessary to test parts of PEP 380 through bytecode structures rather than semantics? Those tests aren't going to be usable by other implementations. The affected tests aren't testing the PEP 380 semantics, they're specifically testing CPython's bytecode generation for yield from expressions and disassembly of same. Just because they aren't of any interest to other implementations doesn't mean *we* don't need them :) There are plenty of behavioural tests to go along with the bytecode specific ones, and those *will* be useful to other implementations. Cheers, Nick. I'm with nick on this one, seems like a very useful test, just remember to mark it as @impl_detail (or however the decorator is called). Fair enough. :-) If other tests are failing (the semantics are wrong) then having a test that shows you the semantics are screwed because the bytecode has been incorrectly generated will be a useful diagnostic tool. On the other hand it is hard to see that bytecode generation could be "wrong" without it affecting some test of semantics that should also fail - so as tests in their own right the bytecode tests *must* be superfluous (or there is some aspect of the semantics that is *only* tested through the bytecode and that seems bad, particularly for other implementations). All the best, Michael Cheers, fijal -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyPy 1.7 - widening the sweet spot
Le 25/11/2011 19:21, Amaury Forgeot d'Arc a écrit : > And oh, I almost forgot distutils, which needs to parse some Makefile which > of course does not exist in PyPy. This is a bug (#10764) that I intend to fix for the next releases of 2.7 and 3.2. I also want to fix all modules that use sys.version[:2] to get 'X.Y', which is a CPython implementation detail. I find PyPy and excellent project, so you can send any bugs in distutils, sysconfig, site and friends my way! I also hope to make distutils2 compatible with PyPy before 2012. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Merging 3.2 to 3.3 is messy because "Misc/NEWS"
Le 26/11/2011 03:18, Jesus Cea a écrit : > On 12/11/11 16:56, Éric Araujo wrote: >> Ezio and I chatted a bit about his on IRC and he may try to write >> a Python parser for Misc/NEWS in order to write a fully automated >> merge tool. > Anything new in this front? :-) Not from me. I don’t have the roundtuits, and I find my hgddmt script good enough. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 404 in (important) documentation in www.python.org and contributor agreement
On Fri, Nov 25, 2011, Jesus Cea wrote: > > Checking documentation abut the contributor license agreement, I had > encounter a wrong HTML link in http://www.python.org/about/help/ : > > * "Python Patch Guidelines" points to > http://www.python.org/dev/patches/, that doesn't exist. Fixed > PS: The devguide doesn't say anything (AFAIK) about the contributor > agreement. The devguide seems to now be hosted on docs.python.org and AFAIK the web team doesn't deal with that. Someone from python-dev needs to lead. -- Aahz ([email protected]) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Merging 3.2 to 3.3 is messy because "Misc/NEWS"
Le 26/11/2011 06:29, Nick Coghlan a écrit : > On Sat, Nov 26, 2011 at 3:14 PM, Raymond Hettinger wrote: >> To me, it would make more sense to split the file into a Misc/NEWS3.2 and >> Misc/NEWS3.3 much as we've done with whatsnew. That would make merging a >> piece of cake and would avoid adding a parser (and its idiosyncracies) to the >> toolchain. > > +1 > > A simple-but-it-works approach to this problem sounds good to me. > > We'd still need to work out a few conventions about how changes that > affect both versions get recorded (I still favour putting independent > entries in both files), but simply eliminating the file name collision > will also eliminate most of the merge conflicts. Maybe I’m not seeing something, but adding an entry by hand does not sound much better than solving conflicts by hand. Another idea: If we had different sections for bug fixes and new features (with or without another level of core/lib/doc/tests separation), then there should be less (no?) conflicts. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Long term development external named branches and periodic merges from python
Le 24/11/2011 22:46, Xavier Morel a écrit : > Wouldn't it be simpler to just use MQ and upload the patch(es) > from the series? MQ is a very powerful and useful tool, but its learning curve is steeper than regular Mercurial, and it is not designed for long-term development. Rebasing patches is more fragile and less user-friendly than merging branches, and it’s also easier to corrupt your MQ patch queue than your Mercurial repo. I like Mercurial merges and I don’t like diffs of diffs, so I avoid MQ. > Would be easier to keep in sync with the development tip too. How so? With a regular clone you have to pull and merge regularly, with MQ you have to pull and rebase. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Long term development external named branches and periodic merges from python
Le 25/11/2011 03:39, Jesus Cea a écrit : > On 24/11/11 18:08, Éric Araujo wrote: >>> I have a question and I would rather have an answer instead of >>> actually trying and getting myself in a messy situation. >> Clones are cheap, trying is cheap! > [snip valid reasons for not trying] My reply was tongue-in-cheek :) FYI, it’s not considered pollution to use a tracker issue to test hooks or Mercurial integration (there’s even one issue entirely devoted to such tests, but I can’t find its number). >>> 5. Development of the new feature is taking a long time, and >>> python canonical version keeps moving forward. The clone+branch >>> and the original python version are diverging. Eventually there >>> are changes in python that the programmer would like in her >>> version, so she does a "pull" and then a merge for the original >>> python branch to her named branch. >> I do this all the time. I work on a fix- branch, and once a >> week for example I pull and merge the base branch. Sometimes there >> are no conflicts except Misc/NEWS, sometimes I have to adapt my >> code because of other people’s changes before I can commit the >> merge. > That is good, because that means your patch is always able to be > applied to the original branch tip, and that you changes work with > current work in the mainline. > > That is what I want to do, but I need to know that it is safe to do so > (from the "Create Patch" perspective). I don’t understand “safe”. >>> 6. What would be posted in the bug tracker when she does a new >>> "Create Patch"?. Only her changes, her changes SINCE the merge, >>> her changes plus merged changes or something else?. >> The diff would be equivalent to “hg diff -r base” and would contain >> all the changes she did to add the bug fix or feature. Merging >> only makes sure that the computed diff does not appear to touch >> unrelated files, IOW that it applies cleanly. (Barring bugs in >> Mercurial-Roundup integration, we have a few of these in the >> metatracker.) > So you are saying that "Create patch" will ONLY get the differences in > the development branch and not the changes brought in from the merge?. I don’t really understand how you understood what I said :( The merge brings in changes from default; when you diff your branch against default later, it will not show the changes brought by the merge, but it will apply cleanly on top of default. Does this wording make sense? Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecation policy
Hi, +1 to all Ezio said. One specific remark: PendingDeprecationWarning could just become an alias of DeprecationWarning, but maybe there is code out there that relies on the distinction, and there is no real value in making it an alias (there is value in removing it altogether, but we can’t do that, can we?). I don’t see the need to deprecate PDW, except in documentation, and am -1 to the metaclass idea (no need). Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
Hi, Going through my email backlog. > Le 11/08/2011 20:30, P.J. Eby a écrit : >> At 04:39 PM 8/11/2011 +0200, Éric Araujo wrote: (By the way, both of these additions to the import protocol (i.e. the dynamically-added ``__path__``, and dynamically-created modules) apply recursively to child packages, using the parent package's ``__path__`` in place of ``sys.path`` as a basis for generating a child ``__path__``. This means that self-contained and virtual packages can contain each other without limitation, with the caveat that if you put a virtual package inside a self-contained one, it's gonna have a really short ``__path__``!) >>> I don't understand the caveat or its implications. >> Since each package's __path__ is the same length or shorter than its >> parent's by default, then if you put a virtual package inside a >> self-contained one, it will be functionally speaking no different >> than a self-contained one, in that it will have only one path >> entry. So, it's not really useful to put a virtual package inside a >> self-contained one, even though you can do it. (Apart form it >> letting you avoid a superfluous __init__ module, assuming it's indeed >> superfluous.) I still don’t understand why this matters or what negative effects it could have on code, but I’m fine with not understanding. I’ll trust that people writing or maintaining import-related tools will agree or complain about that item. >>> I’ll just regret that it's not possible to provide a module docstring >>> to inform that this is a namespace package used for X and Y. >> It *is* possible - you'd just have to put it in a "zc.py" file. IOW, >> this PEP still allows "namespace-defining packages" to exist, as was >> requested by early commenters on PEP 382. It just doesn't *require* >> them to exist in order for the namespace contents to be importable. That’s quite cool. I guess such a namespace-defining module (zc.py here) would be importable, right? Also, would it cause worse performance for other zc.* packages than if there were no zc.py? >>> This was probably said on import-sig, but here I go: yet another import >>> artifact in the sys module! I hope we get ImportEngine in 3.3 to clean >>> up all this. >> Well, I rather *like* having them there, personally, vs. having to >> learn yet another API, but oh well, whatever. Agreed with “whatever” :) I just like to grunt sometimes. >> AFAIK, ImportEngine isn't going to do away with the need for the >> global ones to live somewhere, Yep, but as Nick replied, at least we’ll gain one structure to rule them all. >>> Let's imagine my application Spam has a namespace spam.ext for plugins. >>> To use a custom directory where plugins are stored, or a zip file with >>> plugins (I don't use eggs, so let me talk about zip files here), I'd >>> have to call sys.path.append *and* pkgutil.extend_virtual_paths? >> As written in the current proposal, yes. There was some discussion >> on Python-Dev about having this happen automatically, and I proposed >> that it could be done by making virtual packages' __path__ attributes >> an iterable proxy object, rather than a list: That sounds a bit too complicated. What about just having pkgutil.extend_virtual_paths call sys.path.append? For maximum flexibility, extend_virtual_paths could have an argument to avoid calling sys.path.append. >>> Besides, putting data files in a Python package is held very poorly by >>> some (mostly people following the File Hierarchy Standard), >> ISTM that anybody who thinks that is being inconsistent in >> considering the Python code itself to not be a "data file" by that >> same criterion... especially since one of the more common uses for >> such "data" files are for e.g. HTML templates (which usually contain >> some sort of code) or GUI resources (which are pretty tightly bound >> to the code). A good example is documentation: Having a unique location (/usr/share/doc) for all installed software makes my life easier. Another example is JavaScript files used with HTML documents, such as jQuery: Debian recently split the jQuery file out of their Sphinx package, so that there is only one library installed that all packages can use and that can be updated and fixed once for all. (I’m simplifying; there can be multiple versions of libraries, but not multiple copies. I’ll stop here; I’m not one of the authors of the Filesystem Hierarchy Standard, and I’ll rant against package_data in distutils mailing lists :) >>> A pure virtual package having no source file, I think it should have no >>> __file__ at all. Antoine and someone else thought likewise (I can find the link if you want); do you consider it consensus enough to update the PEP? Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.c
Re: [Python-Dev] Merging 3.2 to 3.3 is messy because "Misc/NEWS"
2011/11/25 Raymond Hettinger : > To me, it would make more sense to split the file into a Misc/NEWS3.2 and > Misc/NEWS3.3 much as we've done with whatsnew. That would make merging a > piece of cake and would avoid adding a parser (an its idiosyncracies) to the > toolchain. Would we not add fixes from 3.2, which were ported to 3.3 to the NEWS3.3 file then? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ImportError: No module named multiarray (is back)
Hi,
I apologize in advance for the length of this mail.
sys.path
When a script or a module is executed by invoking python with proper
arguments, sys.path is extended. When a path to script is given, the
directory containing the script is prepended. When '-m' or '-c' is used,
$CWD is prepended. This is documented in
http://docs.python.org/dev/using/cmdline.html, so far ok.
sys.path and $PYTHONPATH is like $PATH -- if you can convince someone to
put a directory under your control in any of them, you can execute code
as this someone. Therefore, sys.path is dangerous and important.
Unfortunately, sys.path manipulations are only described very briefly,
and without any commentary, in the on-line documentation. python(1)
manpage doesn't even mention them.
The problem: each of the commands below is insecure:
python /tmp/script.py (when script.py is safe by itself)
('/tmp' is added to sys.path, so an attacker can override any
module imported in /tmp/script.py by writing to /tmp/module.py)
cd /tmp && python -mtimeit -s 'import numpy' 'numpy.test()'
(UNIX users are accustomed to being able to safely execute
programs in any directory, e.g. ls, or gcc, or something.
Here '' is added to sys.path, so it is not secure to run
python is other-user-writable directories.)
cd /tmp/ && python -c 'import numpy; print(numpy.version.version)'
(The same as above, '' is added to sys.path.)
cd /tmp && python
(The same as above).
IMHO, if this (long-lived) behaviour is necessary, it should at least be
prominently documented. Also in the manpage.
Prepending realpath(dirname(scriptname))
Before adding a directory to sys.path as described above, Python
actually runs os.path.realpath over it. This means that if the path to a
script given on the commandline is actually a symlink, the directory
containing the real file will be executed. This behaviour is not really
documented (the documentation only says "the directory containing that
file is added to the start of sys.path"), but since the integrity of
sys.path is so important, it should be, IMHO.
Using realpath instead of the (expected) path specified by the user
breaks imports of non-pure-python (mixed .py and .so) modules from
modules executed as scripts on Debian. This is because Debian installs
architecture-independent python files in /usr/share/pyshared, and
symlinks those files into /usr/lib/pymodules/pythonX.Y/. The
architecture-dependent .so and python-version-dependent .pyc files are
installed in /usr/lib/pymodules/pythonX.Y/. When a script, e.g.
/usr/lib/pymodules/pythonX.Y/script.py, is executed, the directory
/usr/share/pyshared is prepended to sys.path. If the script tries to
import a module which has architecture-dependent parts (e.g. numpy) it
first sees the incomplete module in /usr/share/pyshared and fails.
This happens for example in parallel python
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620551) and recently
when packaging CellProfiler for Debian.
Again, if this is on purpose, it should be documented.
PEP 395 (Qualified Names for Modules)
=
PEP 395 proposes another sys.path manipulation. When running a script,
the directory tree will be walked upwards as long as there are
__init__.py files, and then the first directory without will be added.
This is of course a fine idea, but it makes a scenario, which was
previously safe, insecure. More precisely, when executing a script in a
directory in a parent directory-writable-by-other-users, the parent
directory will be added to sys.path.
So the (safe) operation of downloading an archive with a package,
unzipping it in /tmp, changing into the created directory, checking that
the script doesn't do anything bad, and running a script is now insecure
if there is __init__.py in the archive root.
I guess that it would be useful to have an option to turn off those
sys.path manipulations.
Zbyszek
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ImportError: No module named multiarray (is back)
2011/11/27 Zbigniew Jędrzejewski-Szmek : > I guess that it would be useful to have an option to turn off those sys.path > manipulations. Yeah, I recently proposed exactly that (a '--nopath0' option) in http://bugs.python.org/issue13475 (that issue also proposes a "-p/--path0" option to *override* the automatic initialisation of sys.path[0] with a different value). I may still make this general question part of the proposals in PEP 395, though, since it's fairly closely related to many of the issues already discussed by that PEP and is something that will need to be thought out fairly well to make sure it achieves the objective of avoiding cross-user interference. There are limits to what we can do by default due to backwards compatibility concerns, but it should at least be possible to provide better tools to help manage the problem. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
