Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Stephen J. Turnbull wrote: > Andrew Bennetts writes: > > > No, that just means you shouldn't trust *root*. Which is where a > > VM is a very useful tool. You can have the “as root” environment > > for your tests without the need to have anything important trust it. > > Cameron acknowledges that he missed that. So maybe he was right for > the wrong reason; he's still right. But in the current context, it is > not an argument for not worrying, because there is no evidence at all > that the OP set up his buildbot in a secure sandbox. As I read his > followups, he simply "didn't bother" to set up an unprivileged user > and run the 'bot as that user. I made no claim about how the bot was deployed. The point I was disputing was more general than how one specific bot is deployed. To quote the mail I was replying to again: “HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted.” This sentiment was expressed strongly and repeatedly in several mails. It was this overly broad assertion I was addressing, and happily my argument was apparently convincing. I'm fine with “It's not worth running the tests as root because the overhead of making a secure setup for it with a VM etc is too hard with our very limited volunteer resources.” I'm not fine with “We mustn't run them as root because it's impossible to do it safely.” That's all I'm saying. […] > that was *not* the case; the assumption is falsified. Nevertheless, > several people who I would have thought would know better are *all* > arguing from the assumption that the OP configured his test system > with security (rather than convenience) in mind, and are castigating > Cameron for *not* making that same assumption. To my mind, every post > is increasing justification for his unease. :-( I certainly hope I wasn't so severe as to be castigating! If I was Cameron has been kind enough to not show any offense. > And that's why this thread belongs on this list, rather than on Bruce > Schneier's blog. It's very easy these days to set up a basic personal > VM, and folk of goodwill will do so to help the project with buildbots > to provide platform coverage in testing new code. But this > contribution involves certain risks (however low probability, some > Very Bad Things *could* happen). Contributors should get help in > evaluating the potential threats and corresponding risks, and in > proper configuration. Not assurances that nothing will go wrong > "because you probably run the 'bot in a VM." For the record, in case it isn't obvious, I think a buildslave that runs the tests as root that doesn't take precautions like using a VM dedicated to just running the tests (and not running the buildslave) is a bad idea. Although given that there's a very limited supply of volunteer labour involved in configuring and administering buildslaves I'm not surprised to hear this has happened. :( I don't object at all to folks like Cameron asking questions to ensure that these systems are secure enough. I think that's a good thing! I don't even object to treating someone saying “run as root” as a red flag requiring further explanation. What I was objecting to was an apparent willingness to make an unnecessary compromise on software quality. I care about the security of contributors' buildslaves. I also care about the reliability of Python. -Andrew. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Identifier API
In benchmarking PEP 393, I noticed that many UTF-8 decode
calls originate from C code with static strings, in particular
PyObject_CallMethod. Many of such calls already have been optimized
to cache a string object, however, PyObject_CallMethod remains
unoptimized since it requires a char*.
I find the ad-hoc approach of declaring and initializing variables
inadequate, in particular since it is difficult to clean up all
those string objects at interpreter shutdown.
I propose to add an explicit API to deal with such identifiers.
With this API,
tmp = PyObject_CallMethod(result, "update", "O", other);
would be replaced with
PyObject *tmp;
Py_identifier(update);
...
tmp = PyObject_CallMethodId(result, &PyId_update, "O", other);
Py_identifier expands to a struct
typedef struct Py_Identifier {
struct Py_Identifier *next;
const char* string;
PyObject *object;
} Py_Identifier;
string will be initialized by the compiler, next and object on
first use. The new API for that will be
PyObject* PyUnicode_FromId(Py_Identifier*);
PyObject* PyObject_CallMethodId(PyObject*, Py_Identifier*, char*, ...);
PyObject* PyObject_GetAttrId(PyObject*, Py_Identifier*);
int PyObject_SetAttrId(PyObject*, Py_Identifier*, PyObject*);
int PyObject_HasAttrId(PyObject*, Py_Identifier*);
I have micro-benchmarked this; for
import time
d={}
i=d.items()
t=time.time()
for _ in range(10**6):
i | d
print(time.time()-t)
I get a speed-up of 30% (notice that "i | d" invokes
the above PyObject_CallMethod call).
Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Pretending the snark to be slightly serious: you've missed the point. The builtbots are building unreliable code, that being the point of the test suite. Doing unpredictable stuff as root is bad juju. Running the builtbots and their tests should not be run as root except for a very few special tests, and those few need careful consideration and sandboxing. No no no no no. Running as a non-"privileged" user does not gain much. The code may be un*reliable*, but it is not un*trusted*. If the code disturbs the system, it can do so nearly as much as an unprivileged user as the superuser. The critical part of the file system is the build area, and the build slave has full access to that either way. HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted. No, you got that definition wrong. "unreliable" is correct; we don't have any untrusted code in Python. We trust all committers, as do we trust the integrity of the repository server. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3101 (Advanced string formatting) base 36 integer presentation type
Le 08/10/2011 15:03, Antoine Pitrou a écrit : On Fri, 07 Oct 2011 21:14:44 -0600 Jeffrey wrote: I would like to suggest adding an integer presentation type for base 36 to PEP 3101. I can't imagine that it would be a whole lot more difficult than the existing types. Python's built-in long integers provide a nice way to prototype and demonstrate cryptographic operations, especially with asymmetric cryptography. (Alice and Bob stuff.) Built-in functions provide modular reduction, modular exponentiation, and lots of nice number theory stuff that supports a variety of protocols and algorithms. A frequent need is to represent a message by a number. Base 36 provides a way to represent all 26 letters in a semi-standard way, and simple string transformations can efficiently make zeros into spaces or vice versa. Why base 36 rather than, say, base 64 or even base 80? Base 85 is the most efficient base to format IPv6 addresses! http://tools.ietf.org/html/rfc1924 And Python doesn't provide builtin function for this base! Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3101 (Advanced string formatting) base 36 integer presentation type
On Sat, 08 Oct 2011 17:14:55 +0200 Victor Stinner wrote: > Le 08/10/2011 15:03, Antoine Pitrou a écrit : > > On Fri, 07 Oct 2011 21:14:44 -0600 > > Jeffrey wrote: > >> I would like to suggest adding an integer presentation type for base 36 > >> to PEP 3101. I can't imagine that it would be a whole lot more > >> difficult than the existing types. Python's built-in long integers > >> provide a nice way to prototype and demonstrate cryptographic > >> operations, especially with asymmetric cryptography. (Alice and Bob > >> stuff.) Built-in functions provide modular reduction, modular > >> exponentiation, and lots of nice number theory stuff that supports a > >> variety of protocols and algorithms. A frequent need is to represent a > >> message by a number. Base 36 provides a way to represent all 26 letters > >> in a semi-standard way, and simple string transformations can > >> efficiently make zeros into spaces or vice versa. > > > > Why base 36 rather than, say, base 64 or even base 80? > > Base 85 is the most efficient base to format IPv6 addresses! > > http://tools.ietf.org/html/rfc1924 Indeed, I meant base 85. There's a specification that doesn't rely on long integer division: http://en.wikipedia.org/wiki/Ascii85 Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
On Sat, 08 Oct 2011 16:54:06 +0200 "Martin v. Löwis" wrote: > > I find the ad-hoc approach of declaring and initializing variables > inadequate, in particular since it is difficult to clean up all > those string objects at interpreter shutdown. > > I propose to add an explicit API to deal with such identifiers. That sounds like a good idea. > With this API, > > tmp = PyObject_CallMethod(result, "update", "O", other); > > would be replaced with > > PyObject *tmp; > Py_identifier(update); > ... > tmp = PyObject_CallMethodId(result, &PyId_update, "O", other); Surely there is something missing to initialize the "const char *" in the structure? Or is "Py_identifier()" actually a macro? > string will be initialized by the compiler, next and object on > first use. The new API for that will be > > PyObject* PyUnicode_FromId(Py_Identifier*); > PyObject* PyObject_CallMethodId(PyObject*, Py_Identifier*, char*, ...); > PyObject* PyObject_GetAttrId(PyObject*, Py_Identifier*); > int PyObject_SetAttrId(PyObject*, Py_Identifier*, PyObject*); > int PyObject_HasAttrId(PyObject*, Py_Identifier*); Do we want to make these APIs public? Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
Am 08.10.2011 17:43, schrieb Antoine Pitrou: > On Sat, 08 Oct 2011 16:54:06 +0200 > "Martin v. Löwis" wrote: >> >> I find the ad-hoc approach of declaring and initializing variables >> inadequate, in particular since it is difficult to clean up all >> those string objects at interpreter shutdown. >> >> I propose to add an explicit API to deal with such identifiers. > > That sounds like a good idea. > >> With this API, >> >> tmp = PyObject_CallMethod(result, "update", "O", other); >> >> would be replaced with >> >> PyObject *tmp; >> Py_identifier(update); >> ... >> tmp = PyObject_CallMethodId(result, &PyId_update, "O", other); > > Surely there is something missing to initialize the "const char *" in > the structure? Or is "Py_identifier()" actually a macro? Yes (note the parenthesized usage). >> string will be initialized by the compiler, next and object on >> first use. The new API for that will be >> >> PyObject* PyUnicode_FromId(Py_Identifier*); >> PyObject* PyObject_CallMethodId(PyObject*, Py_Identifier*, char*, ...); >> PyObject* PyObject_GetAttrId(PyObject*, Py_Identifier*); >> int PyObject_SetAttrId(PyObject*, Py_Identifier*, PyObject*); >> int PyObject_HasAttrId(PyObject*, Py_Identifier*); > > Do we want to make these APIs public? Probably not at first. I'd suggest making them private for Python 3.3, and if the approach proves satisfying, we can think about adding them to the public API in Python 3.4. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Bring new features to older python versions
Hello everybody, at work we're using different versions of python, from 2.4 to 2.7. Because of the differences between the various versions in terms of features we have a "util.pycompat" module which basically is a copy & paste of different features which were added to stdlib in every new major version throughout the years. What we do is basically this. Instead of: from collections import namedtuple, OrderedDict import fractions ...we do: from util.pycompat.collections import namedtuple, OrderedDict from util.pycompat import fractions # py 2.6 from util.pycompat.builtins import all, any # py 2.5 # etc... This let us use different stdlib features which appeared in latest Python versions (including 3.2) throughout all our code base. Now, what I have in mind is to release this as a public module so that everyone who cannot upgrade to a recent python version can benefit of newer features. By taking a quick look at the various "what's new" documents this is a brief list of what this module would include: functools (2.5) any, all builtins (2.5) collections.defaultdict (2.5) property setters/deleters (2.6) abc (2.6) fractions (2.6) collections.OrderedDict (2.7) collections.Counter (2.7) unittest2 (2.7) functools.lru_cache (3.2) functools.total_ordering (3.2) itertools.accumulate (3.2) reprlib (3.2) contextlib.ContextDecorator (3.2) I have a couple of doubts about this though. The first one is about licensing. What I would be doing is basically copy & paste pieces of the python stdlib modules (including tests) and, where needed, adjust them so that they work with older python versions. Would this represent problem? My second doubt is about morality. Although this might be useful to those people who are forced to use older python versions, on the other hand it might represent an incentive for not upgrading (and there will be python 3.X features as well). Or maybe it won't, I don't know, point is I feel kind of guilty. =) I'd like to hear your opinions, especially about the second point. Best regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
Ciao Giampaolo, > I have a couple of doubts about this though. > The first one is about licensing. > What I would be doing is basically copy & paste pieces of the python > stdlib modules (including tests) and, where needed, adjust them so > that they work with older python versions. > Would this represent problem? I don't think so. Python is distributed under a free non-copyleft license, which means you are basically free to do what you want as long as you don't try to change that license, or misrepresent authorship. (you can also mix that code with code under another license, such as the BSD, the GPL or even a proprietary license) > My second doubt is about morality. > Although this might be useful to those people who are forced to use > older python versions, on the other hand it might represent an > incentive for not upgrading (and there will be python 3.X features as > well). > Or maybe it won't, I don't know, point is I feel kind of guilty. =) I don't know. Certainly we would prefer people to upgrade. Also, the kind of support you will be able to provide as a single maintainer of that package may not be as good as what we collectively provide for the official Python distribution. There's also some stuff there that is coded in C, or that will rely on some functionality of the core interpreter that is not easily emulated on previous versions. But I suppose you'll find that out by yourself. But I wouldn't describe that as "immoral"; rather, suboptimal. cheers Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
The first one is about licensing. What I would be doing is basically copy& paste pieces of the python stdlib modules (including tests) and, where needed, adjust them so that they work with older python versions. Would this represent problem? You have a "nonexclusive, royalty-free, world-wide license to ..." "prepare derivative works, distribute, and otherwise use Python alone or in any derivative version," so: no, this is no problem ... "provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee." My second doubt is about morality. Although this might be useful to those people who are forced to use older python versions, on the other hand it might represent an incentive for not upgrading (and there will be python 3.X features as well). Don't worry about that. I'm not sure how many people would be interested in your approach in the first place - if I have to support old versions of Python, I personally just don't use newer features, and don't even have the desire to do so. If I want to use newer features, I decide to drop support for older versions. That I get both with a hack as such a module is just something that I *personally* would never consider (there are other reasons for me to consider hacks like this, such as when supporting multiple versions is just not feasible, but I wouldn't use a hack for convenience reasons). People that do feel the same way as you have probably started their own emulation layers already, so by publishing your emulation layer, it's not getting worse. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
On Sat, Oct 8, 2011 at 8:35 PM, "Martin v. Löwis" wrote: >> The first one is about licensing. >> What I would be doing is basically copy& paste pieces of the python >> stdlib modules (including tests) and, where needed, adjust them so >> that they work with older python versions. >> Would this represent problem? > > You have a "nonexclusive, royalty-free, world-wide license to ..." > "prepare derivative works, distribute, and otherwise use Python alone or in > any derivative version," so: no, this is no problem ... > > "provided, however, that PSF's License Agreement and PSF's notice of > copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, > 2008, 2009, 2010, 2011 Python Software Foundation; All Rights Reserved" are > retained in Python alone or in any derivative version prepared by Licensee." > >> My second doubt is about morality. >> Although this might be useful to those people who are forced to use >> older python versions, on the other hand it might represent an >> incentive for not upgrading (and there will be python 3.X features as >> well). > > Don't worry about that. I'm not sure how many people would be interested > in your approach in the first place - if I have to support old versions > of Python, I personally just don't use newer features, and don't even > have the desire to do so. If I want to use newer features, I decide to > drop support for older versions. That I get both with a hack as such > a module is just something that I *personally* would never consider > (there are other reasons for me to consider hacks like this, such as when > supporting multiple versions is just not feasible, but I wouldn't > use a hack for convenience reasons). > > People that do feel the same way as you have probably started their > own emulation layers already, so by publishing your emulation layer, > it's not getting worse. > > Regards, > Martin Most programs I know have it's own imperfect version of such thing, so I would definitely use it. Not everyone can drop support for older versions of python at will. Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
I have some similar code in kitchen: http://packages.python.org/kitchen/api-overview.html It wasn't as ambitious as your initial goals sound (I was only working on pulling out what was necessary for what people requested rather than an all-inclusive set of changes). You're welcome to join me and work on this aspect of kitchen if you'd like or you can go your own way and I'll probably start pointing people at your library (Like I do with hashlib, bunch, iterutils, ordereddict, etc). I have a need to support a small amount of code as far back as python-2.3 I don't suppose you're interested in that as well? ;-) On Sat, Oct 08, 2011 at 06:57:47PM +0200, Giampaolo Rodolà wrote: > functools (2.5) > any, all builtins (2.5) > collections.defaultdict (2.5) > property setters/deleters (2.6) > abc (2.6) > fractions (2.6) > collections.OrderedDict (2.7) > collections.Counter (2.7) > unittest2 (2.7) > functools.lru_cache (3.2) > functools.total_ordering (3.2) > itertools.accumulate (3.2) > reprlib (3.2) > contextlib.ContextDecorator (3.2) > You can also add subprocess to this list. There's various methods and functions that were added to subprocess since it's first appearance in python-2.4 (Check the library docs page for notes about this [1] _) hashlib (which has a pypi backport already) is another one. hmac is a third which you probably won't notice if you're just perusing docs. It's an issue because if someone tries to use the stdlib's hmac together with the pypi hashlib, hmac will fail unless it's from a recent enough python. .. [1]_:: http://docs.python.org/library/subprocess.html Speaking as someone who works on a Linux distribution, one thing that I'd appreciate is if you could take care to make it so the copied code doesn't get used if the stdlib already provides the necessary code. If you do this, it makes it easier for people who have to audit the code to do their jobs. Instead of having to check every consumer of the compat library to make sure they use something like this:: try: import functools except ImportError: from pycompat import functools import sys if sys.version_info >= (2, 5): import hmac else: from pycompat import hmac You can depend on roughly the same logic having been performed in the library itself which greatly eases their burden. You can look at the kitchen pycompat code for some examples of doing this [2]_. .. [2]_ http://bzr.fedorahosted.org/bzr/kitchen/devel/files/head:/kitchen/ -Toshio pgpWhofvduZYY.pgp Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
On Sat, Oct 8, 2011 at 21:47, Toshio Kuratomi wrote: > I have some similar code in kitchen: > http://packages.python.org/kitchen/api-overview.html It also sounds similar to six: http://pypi.python.org/pypi/six Avoid all the duplicate efforts would certainly make sense. Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
2011/10/8 Dirkjan Ochtman : > On Sat, Oct 8, 2011 at 21:47, Toshio Kuratomi wrote: >> I have some similar code in kitchen: >> http://packages.python.org/kitchen/api-overview.html > > It also sounds similar to six: > > http://pypi.python.org/pypi/six Though six tries to be a bit minimalist and doesn't strive to include the "Kitchen sink" as it were. :) -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
On 10/8/2011 12:57 PM, Giampaolo Rodolà wrote: I have a couple of doubts about this though. The first one is about licensing. Other have answered -- follow the license in giving credit, etc. My second doubt is about morality. Although this might be useful to those people who are forced to use older python versions, on the other hand it might represent an incentive for not upgrading (and there will be python 3.X features as well). Or maybe it won't, I don't know, point is I feel kind of guilty. =) I'd like to hear your opinions, especially about the second point. While I would personally prefer that everyone else upgrade, I never expected that and I consider it a feature that one does not have to. The only thing that has really bothered me is people spreading FUD about Python 3, and that has mostly ended. Just be accurate in promoting your package. It will backport some newer stdlib features, but it will not remove old cruft, fix bugs, add other new features, or include doc improvements. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
Hi, > abc (2.6) I’m not sure this module is very useful without built-in support in isinstance and issubclass. > collections.OrderedDict (2.7) > unittest2 (2.7) Why not depend on the backports available on PyPI instead of re-backporting these in your project? > My second doubt is about morality. > Although this might be useful to those people who are forced to use > older python versions, on the other hand it might represent an > incentive for not upgrading (and there will be python 3.X features as > well). It’s more about marketing than morality IMO :) As other people have said, many projects already have manual backports, so converging efforts on six (for a minimal compat layer) or your lib (for a fat layer) is just rationalization of existing practices. New versions of Python can fend for themselves IMO, they’re not threatened that much by one lib with backports. The issues I foresee with your lib are more technical: First, it looks like a big bag of backported modules, classes and functions without defined criterion for inclusion (“cool new stuff”?). Second, will you keep on lumping new things until Python 3.4? 3.8? Won’t that become unmanageable (boring/huge/hard)? Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
