[Python-Dev] Release manager/developer (10 years + experience) would like to help and volunteer time if needed
Hi, I have worked 10 years at Adobe Systems as a Release Developer for the LiveCycle ES team and am now employed as a Release Manager (for a team of one, me ;) ) at Nuance Communications since last March. I have put lots of effort to keep Python alive and well at Adobe by providing complete build/release solutions & processes, automation and tooling in my favourite language, Python. I have been promoting, planning and implementing a completely new build/release infrastructure at Nuance, where my expectation is have a 100% python shop to manage builds and releases. I would very like to offer any help you may require, provided I am a good fit. I can provide references, resume, etc. if requested. In hopes of pursuing further discussions, please accept my best regards, Martin Senecal Gatineau (Quebec) Canada___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release manager/developer (10 years + experience) would like to help and volunteer time if needed
Hi Nick, That sounds great! I assume since python-dev has been cc'ed that both Martin von Loewis ans Dirkjan Ochtman are listening on this thread. If so, then let me know if there is anything specific that either of you would need a hand with. I would be more than happy to take on some of your "still TODO but no time" items. Meanwhile I will take a closer look @ http://www.python.org/dev/intro and see where/if I can roll up my sleeves and lend a hand. Thanks for the reply & info and I look forward to contributing! Mart :) On 2010-06-16, at 9:19 AM, Nick Coghlan wrote: > On Wed, Jun 16, 2010 at 3:45 PM, Mart wrote: >> >> I have put lots of effort to keep Python alive and well at Adobe by >> providing complete build/release solutions & processes, automation and >> tooling in my favourite language, Python. >> I have been promoting, planning and implementing a completely new >> build/release infrastructure at Nuance, where my expectation is have a 100% >> python shop to manage builds and releases. > > Hi Martin, > > With that kind of background there are likely a number of ways you > could contribute. From a general Python programming point of view, I'd > start with Brett's intro to CPython development at > http://www.python.org/dev/intro/ and the other links in the dev > section of the web site. There are plenty of bug fixes and feature > requests relating to pure Python components of the standard library > that always need work (even comments just saying "I tried this patch > and it worked for me" can be very helpful). > > Specifically in the area of automated build and test management, > Martin von Loewis may have some suggestions for improvements that > could be made to our Buildbot infrastructure that he doesn't have the > time to do himself. It may also be worth checking with Dirkjan Ochtman > to see if there is anything in this space that still needs to be > handled for the transition from svn to hg that will hopefully be > happening later this year. With any luck, those two will actually > chime in here (as they're both python-dev subscribers). > > We don't go in for automated binary releases for a variety of reasons > - I definitely advise trawling through the python-dev archives for a > while before getting too enthusiastic on that particular front. > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecated __cmp__ and total ordering
__cmp__ used to provide a convenient way to make all ordering operators work by defining a single method. For better or worse, it's gone in 3.0. To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: class TotalOrderMixin(object): def __lt__(self, other): raise NotImplemented # override this def __gt__(self, other): return other < self def __le__(self, other): return not (other < self) def __ge__(self, other): return not (self < other) __eq__ and __ne__ are somewhat special, although it is possible to define them in terms of __lt__ def __eq__(self, other): return not (self == other) def __ne__(self, other): return self < other or other < self it may be inefficient. So, to avoid the situation where all the projects that match http://www.google.com/codesearch?q=__cmp__+lang%3Apython have to implement their own TotalOrderMixin, perhaps one could be provided in the stdlib? Or even better, shouldn't a class grow automagic __gt__, __le__, __ge__ if __lt__ is provided, and, in a similar vein, __ne__ if __eq__ is provided? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
On Tue, Mar 10, 2009 at 3:57 PM, Michael Foord wrote: > Is there something you don't like about this one: > http://code.activestate.com/recipes/576529/ > Yes -- it is not in the standard library. As I said, eventually all the 15,000 matches on Google Code need to update their code and copy that snippet to their util/, write tests for it etc. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] version compare function into main lib
See http://wiki.python.org/moin/ApplicationInfrastructure , "Version handling" below for a possible strict version API. The page is relevant for the general packaging discussion as well, although it's not fully fleshed out yet. MS On Fri, Mar 27, 2009 at 5:11 PM, "Martin v. Löwis" wrote: > Correct me if I wrong, but shouldn't Python include function for >> version comparisons? >> > > On the packaging summit yesterday, people agreed that yes, we should > have something like that in the standard library, and it should be more > powerful than what distutils currently offers. > > There was no conclusion of how specifically that functionality should > be offered; several people agreed that Python should mandate a standard > format, which it is then able to compare. So you might not be able to > spell it "10.3.40-beta", but perhaps "10.3.40b1" or "10.3.40~beta". > > Regards, > Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] version compare function into main lib
> Instead of trying to parse some version string, distutils should > require defining the version as tuple with well-defined entries - > much like what we have in sys.version_info for Python. > > The developer can then still use whatever string format s/he wants. > > The version compare function would then work on this version tuple > and probably be called cmp() (at least in Python 2.x ;-). > Except there need to be functions for parsing the tuple from a string and preferably a canonical string representation to ease that parsing. Hence the Version class in "Version Handling" referred to above. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] version compare function into main lib
On Sat, Mar 28, 2009 at 12:37 AM, Ben Finney < [email protected] >wrote: > "Martin v. Löwis" writes: > > > I don't mind the setuptools implementation being used as a basis > > (assuming it gets contributed), but *independently* I think a > > specfication is needed what version strings it actually understands. > > Such specification must precede the actual implementation (in > > distutils). > > Yes, please. The comparison of version strings needs to be easily done > by non-Python programs (e.g. tools for packaging Python > distributions), so a specification that can be implemented in other > languages or environments is a must. There's a specification in http://wiki.python.org/moin/ApplicationInfrastructure , see "Version API" below (at least, it's a start). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
The general consensus in python-ideas is that the following is needed, so I
bring it to python-dev to final discussions before I file a feature request
in bugs.python.org.
Proposal: add add_query_params() for appending query parameters to an URL to
urllib.parse and urlparse.
Implementation:
http://github.com/mrts/qparams/blob/83d1ec287ec10934b5e637455819cf796b1b421c/qparams.py(feel
free to fork and comment).
Behaviour (longish, guided by "simple things are simiple, complex things
possible"):
In the simplest form, parameters can be passed via keyword arguments:
>>> add_query_params('foo', bar='baz')
'foo?bar=baz'
>>> add_query_params('http://example.com/a/b/c?a=b', b='d')
'http://example.com/a/b/c?a=b&b=d'
Note that '/', if given in arguments, is encoded:
>>> add_query_params('http://example.com/a/b/c?a=b', b='d', foo='/bar')
'http://example.com/a/b/c?a=b&b=d&foo=%2Fbar'
Duplicates are discarded:
>>> add_query_params('http://example.com/a/b/c?a=b', a='b')
'http://example.com/a/b/c?a=b'
>>> add_query_params('http://example.com/a/b/c?a=b&c=q', a='b', b='d',
... c='q')
'http://example.com/a/b/c?a=b&c=q&b=d'
But different values for the same key are supported:
>>> add_query_params('http://example.com/a/b/c?a=b', a='c', b='d')
'http://example.com/a/b/c?a=b&a=c&b=d'
Pass different values for a single key in a list (again, duplicates are
removed):
>>> add_query_params('http://example.com/a/b/c?a=b', a=('q', 'b', 'c'),
... b='d')
'http://example.com/a/b/c?a=b&a=q&a=c&b=d'
Keys with no value are respected, pass ``None`` to create one:
>>> add_query_params('http://example.com/a/b/c?a', b=None)
'http://example.com/a/b/c?a&b'
But if a value is given, the empty key is considered a duplicate (i.e. the
case of a&a=b is considered nonsensical):
>>> add_query_params('http://example.com/a/b/c?a', a='b', c=None)
'http://example.com/a/b/c?a=b&c'
If you need to pass in key names that are not allowed in keyword arguments,
pass them via a dictionary in second argument:
>>> add_query_params('foo', {"+'|äüö": 'bar'})
'foo?%2B%27%7C%C3%A4%C3%BC%C3%B6=bar'
Order of original parameters is retained, although similar keys are grouped
together. Order of keyword arguments is not (and can not be) retained:
>>> add_query_params('foo?a=b&b=c&a=b&a=d', a='b')
'foo?a=b&a=d&b=c'
>>> add_query_params('http://example.com/a/b/c?a=b&q=c&e=d',
... x='y', e=1, o=2)
'http://example.com/a/b/c?a=b&q=c&e=d&e=1&x=y&o=2'
If you need to retain the order of the added parameters, use an
:class:`OrderedDict` as the second argument (*params_dict*):
>>> from collections import OrderedDict
>>> od = OrderedDict()
>>> od['xavier'] = 1
>>> od['abacus'] = 2
>>> od['janus'] = 3
>>> add_query_params('http://example.com/a/b/c?a=b', od)
'http://example.com/a/b/c?a=b&xavier=1&abacus=2&janus=3'
If both *params_dict* and keyword arguments are provided, values from the
former are used before the latter:
>>> add_query_params('http://example.com/a/b/c?a=b', od, xavier=1.1,
... zorg='a', alpha='b', watt='c', borg='d')
'
http://example.com/a/b/c?a=b&xavier=1&xavier=1.1&abacus=2&janus=3&zorg=a&borg=d&watt=c&alpha=b
'
Do nothing with a single argument:
>>> add_query_params('a')
'a'
>>> add_query_params('arbitrary strange stuff?öäüõ*()+-=42')
'arbitrary strange stuff?\xc3\xb6\xc3\xa4\xc3\xbc\xc3\xb5*()+-=42'
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
On Sun, Apr 12, 2009 at 3:23 PM, Jacob Holm wrote:
> Hi Mart
>
>>>> add_query_params('http://example.com/a/b/c?a=b', b='d', foo='/bar')
>>'http://example.com/a/b/c?a=b&b=d&foo=%2Fbar <
>> http://example.com/a/b/c?a=b&b=d&foo=%2Fbar>'
>>
>> Duplicates are discarded:
>>
>
> Why discard duplicates? They are valid and have a well-defined meaning.
The bad thing about reasoning about query strings is that there is no
comprehensive documentation about their meaning. Both RFC 1738 and RFC 3986
are rather vague in that matter. But I agree that duplicates actually have a
meaning (an ordered list of identical values), so I'll remove the bits that
prune them unless anyone opposes (which I doubt).
>> But if a value is given, the empty key is considered a duplicate (i.e. the
>> case of a&a=b is considered nonsensical):
>>
>
> Again, it is a valid url and this will change its meaning. Why?
I'm uncertain whether a&a=b has a meaning, but don't see any harm in
supporting it, so I'll add the feature.
>>>>> add_query_params('http://example.com/a/b/c?a', a='b', c=None)
>>'http://example.com/a/b/c?a=b&c <http://example.com/a/b/c?a=b&c>'
>>
>> If you need to pass in key names that are not allowed in keyword
>> arguments,
>> pass them via a dictionary in second argument:
>>
>>>>> add_query_params('foo', {"+'|äüö": 'bar'})
>>'foo?%2B%27%7C%C3%A4%C3%BC%C3%B6=bar'
>>
>> Order of original parameters is retained, although similar keys are
>> grouped
>> together.
>>
>
> Why the grouping? Is it a side effect of your desire to discard
> duplicates? Changing the order like that changes the meaning of the url.
> A concrete case where the order of field names matters is the ":records"
> converter in http://pypi.python.org/pypi/zope.httpform/1.0.1 (a small
> independent package extracted from the form handling code in zope).
It's also related to duplicate handling, but it mostly relates to the data
structure used in the initial implementation (an OrderedDict). Re-grouping
is removed now and not having to deal with duplicates simplified the code
considerably (using a simple list of key-value tuples now).
If you change it to keep duplicates and not unnecessarily mangle the field
> order I am +1, else I am -0.
Thanks for your input! Changes pushed to github (see the updated behaviour
there as well):
http://github.com/mrts/qparams/blob/4f32670b55082f8d0ef01c33524145c3264c161a/qparams.py
MS
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
On Mon, Apr 13, 2009 at 12:56 AM, Antoine Pitrou wrote:
> Mart Sõmermaa gmail.com> writes:
> >
> > Proposal: add add_query_params() for appending query parameters to an URL
> to
> urllib.parse and urlparse.
>
> Is there anything to /remove/ a query parameter?
I'd say this is outside the scope of add_query_params().
As for the duplicate handling, I've implemented a threefold strategy that
should address all use cases raised before:
def add_query_params(*args, **kwargs):
"""
add_query_parms(url, [allow_dups, [args_dict, [separator]]], **kwargs)
Appends query parameters to an URL and returns the result.
:param url: the URL to update, a string.
:param allow_dups: if
* True: plainly append new parameters, allowing all duplicates
(default),
* False: disallow duplicates in values and regroup keys so that
different values for the same key are adjacent,
* None: disallow duplicates in keys -- each key can have a single
value and later values override the value (like dict.update()).
:param args_dict: optional dictionary of parameters, default is {}.
:param separator: either ';' or '&', the separator between key-value
pairs, default is '&'.
:param kwargs: parameters as keyword arguments.
:return: original URL with updated query parameters or the original URL
unchanged if no parameters given.
"""
The commit is
http://github.com/mrts/qparams/blob/b9bdbec46bf919d142ff63e6b2b822b5d57b6f89/qparams.py
extensive description of the behaviour is in the doctests.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
On Mon, Apr 13, 2009 at 8:23 PM, Steven Bethard
wrote:
>
> On Mon, Apr 13, 2009 at 2:29 AM, Mart Sõmermaa wrote:
> >
> >
> > On Mon, Apr 13, 2009 at 12:56 AM, Antoine Pitrou
> > wrote:
> >>
> >> Mart Sõmermaa gmail.com> writes:
> >> >
> >> > Proposal: add add_query_params() for appending query parameters to an
> >> > URL to
> >> urllib.parse and urlparse.
> >>
> >> Is there anything to /remove/ a query parameter?
> >
> > I'd say this is outside the scope of add_query_params().
> >
> > As for the duplicate handling, I've implemented a threefold strategy that
> > should address all use cases raised before:
> >
> > def add_query_params(*args, **kwargs):
> > """
> > add_query_parms(url, [allow_dups, [args_dict, [separator]]], **kwargs)
> >
> > Appends query parameters to an URL and returns the result.
> >
> > :param url: the URL to update, a string.
> > :param allow_dups: if
> > * True: plainly append new parameters, allowing all duplicates
> > (default),
> > * False: disallow duplicates in values and regroup keys so that
> > different values for the same key are adjacent,
> > * None: disallow duplicates in keys -- each key can have a single
> > value and later values override the value (like dict.update()).
>
> Unnamed flag parameters are unfriendly to the reader. If I see something like:
>
> add_query_params(url, True, dict(a=b, c=d))
>
> I can pretty much guess what the first and third arguments are, but I
> have no clue for the second. Even if I have read the documentation
> before, I may not remember whether the middle argument is "allow_dups"
> or "keep_dups".
Keyword arguments are already used for specifying the arguments to the
query, so naming can't be used. Someone may need an 'allow_dups' key
in their query and forget to pass it in params_dict.
A default behaviour should be found that works according to most
user's expectations so that they don't need to use the positional
arguments generally.
Antoine Pitrou wrote:
> You could e.g. rename the function to update_query_params() and decide that
> every parameter whose specified value is None must atcually be removed from
> the URL.
I agree that removing parameters is useful. Currently, None is used
for signifying a key with no value. Instead, booleans could be used:
if a key is True (but obviously not any other value that evaluates to
True), it is a key with no value, if False (under the same evaluation
restriction), it should be removed from the query if present. None
should not be treated specially under that scheme. As an example:
>>> update_query_params('http://example.com/?q=foo', q=False, a=True, b='c',
>>> d=None)
'http://example.com/?a&b=c&d=None'
However,
1) I'm not sure about the implications of 'foo is True', I have never
used it and PEP 8 explicitly warns against it -- does it work
consistently across different Python implementations? (Assuming on the
grounds that True should be a singleton no different from None that it
should work.)
2) the API gets overly complicated -- as per the complaint above, it's
usability-challenged already.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
On Sat, Apr 18, 2009 at 3:41 PM, Nick Coghlan wrote: > Yep - Guido has pointed out in a few different API design discussions > that a boolean flag that is almost always set to a literal True or False > is a good sign that there are two functions involved rather than just > one. There are exceptions to that guideline (e.g. the reverse argument > for sorted and list.sort), but they aren't common, and even when they do > crop up, making them keyword-only arguments is strongly recommended. As you yourself previously noted -- "it is often better to use *args for the two positional arguments - it avoids accidental name conflicts between the positional arguments and arbitrary keyword arguments" -- kwargs may cause name conflicts. But I also agree, that the current proliferation of positional args is ugly. add_query_params_no_dups() would be suboptimal though, as there are currently three different ways to handle the duplicates: * allow duplicates everywhere (True), * remove duplicate *values* for the same key (False), * behave like dict.update -- remove duplicate *keys*, unless explicitly passed a list (None). (See the documentation at http://github.com/mrts/qparams/blob/bf1b29ad46f9d848d5609de6de0bfac1200da310/qparams.py ). Additionally, as proposed by Antoine Pitrou, removing keys could be implemented. It feels awkward to start a PEP for such a marginal feature, but clearly a couple of enlightened design decisions are required. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
On Sun, Apr 19, 2009 at 2:06 AM, Nick Coghlan wrote: > That said, I'm starting to wonder if an even better option may be to > just drop the kwargs support from the function and require people to > always supply a parameters dictionary. That would simplify the signature > to the quite straightforward: > > def add_query_params(url, params, allow_dups=True, sep='&') That's the most straightforward and I like this more than the one below. > I agree that isn't a good option, but mapping True/False/None to those > specific behaviours also seems rather arbitrary (specifically, it is > difficult to remember which of "allow_dups=False" and "allow_dups=None" > means to ignore any duplicate keys and which means to ignore only > duplicate items). I'd say it's less of a problem when using named arguments, i.e. you read it as: allow_dups=True : yes allow_dups=False : effeminately no :), allow_dups=None : strictly no which more or less corresponds to the behaviour. > It also doesn't provide a clear mechanism for > extension (e.g. what if someone wanted duplicate params to trigger an > exception?) > > Perhaps the extra argument should just be a key/value pair filtering > function, and we provide functions for the three existing behaviours > (i.e. allow_duplicates(), ignore_duplicate_keys(), > ignore_duplicate_items()) in the urllib.parse module. This would be the most flexible and conceptually right (ye olde strategy pattern), but would clutter the API. > Note that your implementation and docstring currently conflict with each > other - the docstring says "pass them via a dictionary in second > argument:" but the dictionary is currently the third argument (the > docstring also later refers to passing OrderedDictionary as the second > argument). It's a mistake that exemplifies once again that positional args are awkward :). --- So, gentlemen, either def add_query_params(url, params, allow_dups=True, sep='&') or def allow_duplicates(...) def remove_duplicate_values(...) ... def add_query_params(url, params, strategy=allow_duplicates, sep='&') ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wordcode-based Python
ver time: http://tinyurl.com/yh7vmlh slowunpickle: Min: 0.411744 -> 0.356784: 15.40% faster Avg: 0.444638 -> 0.393261: 13.06% faster Significant (t=7.009269, a=0.95) Stddev: 0.04147 -> 0.06044: 31.38% larger Mem max: 7132.000 -> 7848.000: 9.12% larger Usage over time: http://tinyurl.com/yfwvz3g startup_nosite: Min: 0.664456 -> 0.598770: 10.97% faster Avg: 0.933034 -> 0.761228: 22.57% faster Significant (t=20.660776, a=0.95) Stddev: 0.09645 -> 0.06728: 43.37% smaller Mem max: 1940.000 -> 1940.000: -0.00% smaller Usage over time: http://tinyurl.com/yzzxcmd threaded_count: Min: 0.220059 -> 0.138708: 58.65% faster Avg: 0.232347 -> 0.156120: 48.83% faster Significant (t=23.804797, a=0.95) Stddev: 0.01889 -> 0.02586: 26.96% larger Mem max: 6460.000 -> 7664.000: 15.71% larger Usage over time: http://tinyurl.com/yzm3awu unpack_sequence: Min: 0.000129 -> 0.000120: 7.57% faster Avg: 0.000218 -> 0.000194: 12.14% faster Significant (t=3.946194, a=0.95) Stddev: 0.00139 -> 0.00128: 8.13% smaller Mem max: 18948.000 -> 19056.000: 0.57% larger Usage over time: http://tinyurl.com/yf8es3f unpickle: Min: 1.191468 -> 1.206198: 1.22% slower Avg: 1.248471 -> 1.281957: 2.61% slower Significant (t=-2.658526, a=0.95) Stddev: 0.05513 -> 0.11325: 51.32% larger Mem max: 7776.000 -> 8676.000: 10.37% larger Usage over time: http://tinyurl.com/yz96gw2 unpickle_list: Min: 0.922200 -> 0.861167: 7.09% faster Avg: 0.955964 -> 0.976829: 2.14% slower Not significant Stddev: 0.04374 -> 0.21061: 79.23% larger Mem max: 6820.000 -> 8324.000: 18.07% larger Usage over time: http://tinyurl.com/yjbraxg --- The diff between the two trees is at http://dpaste.org/RpIv/ Best, Mart Sõmermaa ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wordcode-based Python
On Wed, Nov 4, 2009 at 5:54 PM, Collin Winter wrote: > Do note that the --track_memory option to perf.py imposes some > overhead that interferes with the performance figures. Thanks for the notice, without -m/--track_memory the deviation in results is indeed much smaller. > I'd recommend > running the benchmarks again without --track_memory. Done: $ python unladen-tests/perf.py -r --benchmarks=-2to3,all py261/python wpy/python Report on Linux zeus 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64 Total CPU cores: 2 ai: Min: 0.629343 -> 0.576259: 9.21% faster Avg: 0.634689 -> 0.581551: 9.14% faster Significant (t=39.404870, a=0.95) Stddev: 0.01259 -> 0.00484: 160.04% smaller call_simple: Min: 1.796710 -> 1.700046: 5.69% faster Avg: 1.801533 -> 1.716367: 4.96% faster Significant (t=137.452069, a=0.95) Stddev: 0.00522 -> 0.00333: 56.64% smaller django: Min: 1.280840 -> 1.275350: 0.43% faster Avg: 1.287179 -> 1.287233: 0.00% slower Not significant Stddev: 0.01055 -> 0.00581: 81.60% smaller iterative_count: Min: 0.211744 -> 0.123271: 71.77% faster Avg: 0.213148 -> 0.128596: 65.75% faster Significant (t=88.510311, a=0.95) Stddev: 0.00233 -> 0.00926: 74.80% larger normal_startup: Min: 0.520829 -> 0.516412: 0.86% faster Avg: 0.559170 -> 0.554678: 0.81% faster Not significant Stddev: 0.02031 -> 0.02093: 2.98% larger pickle: Min: 1.988127 -> 1.926643: 3.19% faster Avg: 2.000676 -> 1.936185: 3.33% faster Significant (t=36.712505, a=0.95) Stddev: 0.01650 -> 0.00603: 173.67% smaller pickle_dict: Min: 1.681116 -> 1.619192: 3.82% faster Avg: 1.701952 -> 1.629548: 4.44% faster Significant (t=34.513963, a=0.95) Stddev: 0.01721 -> 0.01200: 43.46% smaller pickle_list: Min: 0.918128 -> 0.884967: 3.75% faster Avg: 0.925534 -> 0.891200: 3.85% faster Significant (t=60.451407, a=0.95) Stddev: 0.00496 -> 0.00276: 80.00% smaller pybench: Min: 58692 -> 51128: 14.79% faster Avg: 59914 -> 52316: 14.52% faster regex_compile: Min: 0.894190 -> 0.816447: 9.52% faster Avg: 0.900353 -> 0.826003: 9.00% faster Significant (t=24.974080, a=0.95) Stddev: 0.00448 -> 0.02943: 84.78% larger regex_effbot: Min: 0.124442 -> 0.123750: 0.56% faster Avg: 0.134908 -> 0.126137: 6.95% faster Significant (t=5.496357, a=0.95) Stddev: 0.01581 -> 0.00218: 625.68% smaller regex_v8: Min: 0.132730 -> 0.143494: 7.50% slower Avg: 0.134287 -> 0.147387: 8.89% slower Significant (t=-40.654627, a=0.95) Stddev: 0.00108 -> 0.00304: 64.34% larger rietveld: Min: 0.754050 -> 0.737335: 2.27% faster Avg: 0.770227 -> 0.754642: 2.07% faster Significant (t=7.547765, a=0.95) Stddev: 0.01434 -> 0.01486: 3.49% larger slowpickle: Min: 0.858494 -> 0.795162: 7.96% faster Avg: 0.862350 -> 0.799479: 7.86% faster Significant (t=133.690989, a=0.95) Stddev: 0.00394 -> 0.00257: 52.92% smaller slowspitfire: Min: 0.955587 -> 0.909843: 5.03% faster Avg: 0.965960 -> 0.925845: 4.33% faster Significant (t=16.351067, a=0.95) Stddev: 0.01237 -> 0.02119: 41.63% larger slowunpickle: Min: 0.409312 -> 0.346982: 17.96% faster Avg: 0.412381 -> 0.349148: 18.11% faster Significant (t=242.889869, a=0.95) Stddev: 0.00198 -> 0.00169: 17.61% smaller startup_nosite: Min: 0.195620 -> 0.194328: 0.66% faster Avg: 0.230811 -> 0.238523: 3.23% slower Significant (t=-3.869944, a=0.95) Stddev: 0.01932 -> 0.02052: 5.87% larger threaded_count: Min: 0.222133 -> 0.133764: 66.06% faster Avg: 0.236670 -> 0.147750: 60.18% faster Significant (t=57.472693, a=0.95) Stddev: 0.01317 -> 0.00813: 61.98% smaller unpack_sequence: Min: 0.000129 -> 0.000119: 8.43% faster Avg: 0.000132 -> 0.000123: 7.22% faster Significant (t=24.614061, a=0.95) Stddev: 0.3 -> 0.00011: 77.02% larger unpickle: Min: 1.191255 -> 1.149132: 3.67% faster Avg: 1.218023 -> 1.162351: 4.79% faster Significant (t=21.222711, a=0.95) Stddev: 0.02242 -> 0.01362: 64.54% smaller unpickle_list: Min: 0.880991 -> 0.965611: 8.76% slower Avg: 0.898949 -> 0.985231: 8.76% slower Significant (t=-17.387537, a=0.95) Stddev: 0.04838 -> 0.01103: 338.79% smaller ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] CVE tracking
Hello! Does someone systematically track the CVE vulnerability list? Ideally, Python security officers would have close collaboration with whoever manages CVE (like distribution security officers do), so that * every CVE issue would have a corresponding ticket on Python bug tracker (perhaps the process can be automated to some degree?) * that ticket would be referred to in CVE vulnerability page "References" section (see e.g. http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-2315 , that does not have a corresponding Python bug tracker link) * all CVE issues would be listed in http://www.python.org/news/security/ with corresponding information about when the fix has been or will be commited and which upcoming or past release incorporates it. Some relevant links: http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=python http://secunia.com/advisories/product/14172/?task=advisories ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] CVE tracking
I created a script that parses the
http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=python
Python-related CVE list and classifies the CVEs as follows:
* "ok" -- CVE has references to bugs.python.org
* "warnings" -- CVE has references to Python SVN revisions
or an issue in bugs.python.org refers to it (i.e. the probelm is
probably fixed, but the CVE should really be updated to link
to the issue that is probably listed in bugs.python.org)
* "errors" -- CVE does have no references to Python issues or SVN
nor does any issue in bugs.python.org have references to the CVE id
The script is at http://dpaste.com/hold/92930/
The results are at http://dpaste.com/hold/92929/
There were 35 errors, 8 warnings, 5 CVEs were OK.
In an ideal world, the references would be symmetric, i.e. every
Python-related CVE would have references to one or more issues in
bugs.python.org and these issues would also refer back to the CVE id.
###
As for the rmtree problem that Gisle Aas raised, this seems to apply
as of Python 2.6:
---
# emulate removing /etc
$ sudo cp -a /etc /root/etc/
$ sudo python2.6
>>> for i in xrange(0, 5):
... with open("/root/etc/" + str(i), "w") as f:
... f.write("0")
...
$ ls /root/etc > orig_list.txt
$ mkdir /tmp/attack
$ cp -a /root/etc/* /tmp/attack
$ sudo python2.6
>>> from shutil import rmtree
>>> rmtree('/tmp/attack')
>>> # press ctrl-z to suspend execution
^Z
[1]+ Stopped sudo python2.6
$ mv /tmp/attack /tmp/dummy; ln -s /root/etc /tmp/attack
$ fg
sudo python2.6
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.6/shutil.py", line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/local/lib/python2.6/shutil.py", line 223, in rmtree
os.rmdir(path)
OSError: [Errno 20] Not a directory: '/tmp/attack'
$ ls /root/etc > new_list.txt
$ diff -q orig_list.txt new_list.txt
Files orig_list.txt and new_list.txt differ
---
If the attack weren't possible, the lists would not differ.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] CVE tracking
When I looked through that list a week or so ago, I noticed that some issues were obviously related to the Python distribution itself, but others were appeared to be Python application problems. I looked through the list now and weeded out irrelevant CVEs (by putting them into the ignore list in the script). Also, now the output has descriptions of the CVEs as well, so it's more readable. Improved output: http://dpaste.com/hold/93386/ Improved script (with a proper IGNORED_LIST): http://dpaste.com/hold/93388/ The results are much better: 5 OK's, 8 WARNings, 7 ERRORs. Most of the errors are from 2007 or before, the only error from 2008 is an obscure Tools/faqwiz/move-faqwiz.sh-related one. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __import__ problems
Python programmers need to dynamically load submodules instead of
top-level modules -- given a string with module hierarchy, e.g.
'foo.bar.baz', access to the tail module 'baz' is required instead
of 'foo'.
Currently, the common hack for that is to use
modname = 'foo.bar.baz' mod = __import__(modname, {}, {}, [''])
This, however, is indeed an undocumented hack and, what's worse,
causes 'baz' to be imported twice, as 'baz' and 'baz.' (with tail
dot). The problem is reported in [1] and the idiom pops up in about
2000 (sic!) lines in Google Code Search [2].
There at least two workarounds:
* the getattr approach documented in [3]
* use __import__(modname, {}, {}, [modname.rsplit(".", 1)[-1]])
As both of them are clumsy and inefficient, I created a simple patch
for __import__ [4] that adds yet another argument, 'submodule'
(maybe 'tail_module' would have been more appropriate) that caters
for that particular use case:
__import__('foo.bar.baz') # submodule=False by default
__import__('foo.bar.baz', submodule=True)
__import__('foo.bar.baz', fromlist=['baz'])
---
While I was doing that, I noticed that the import_module_level()
function that does the gruntwork behind __import__ does not entirely
match the documentation [3].
Namely, [3] states "the statement from spam.ham import eggs results in
__import__('spam.ham', globals(), locals(), ['eggs'], -1)."
This is incorrect:
__import__('foo.bar', globals(), locals(), ['baz'], -1)
i.e. 'bar' is imported, not 'baz' (or 'ham' and not 'eggs').
As a matter of fact, anything can be in 'fromlist' (the reason for
the API abuse seen in [2]):
__import__('foo.bar.baz', globals(), locals(),
... ['this_is_a_bug'], -1)
So, effectively, 'fromlist' is already functioning as a boolean that
indicates whether the tail or toplevel module is imported.
Proposal:
* either fix __import__ to behave as documented:
# from foo.bar import baz
>>> __import__('foo.bar', fromlist=['baz'])
# from foo.bar import baz, baq
>>> __import__('foo.bar', fromlist=['baz', 'baq'])
(,
)
and add the 'submodule' argument to support the common
__import__ use case [4],
* or if that is not feasible, retain the current boolean behaviour
and make that explicit by renaming 'fromlist' to 'submodule' (and
require the latter to be a boolean, not a list).
Too bad I couldn't come up with this before, 3.0 would have been a
perfect opportunity to get things right (one way or the other).
---
References:
[1] http://bugs.python.org/issue2090
[2] http://google.com/codesearch?hl=en&lr=&q=__import__.*%5C%5B%27%27%5C%5D
[3] http://docs.python.org/library/functions.html#__import__
[4] http://bugs.python.org/issue4438
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __import__ problems
Nick Coghlan wrote:
i.e.
"from foo.bar import baz" >
= __import__('foo.bar', globals(), locals(), ['baz'], -1)
baz = .baz
When there are multiple names being imported or an 'as' clause is
involved, I hope the reasons for doing it this way become more obvious:
"from foo.bar import baz, bob" >
= __import__('foo.bar', globals(), locals(), ['baz', 'bob'], -1)
baz = .baz
bob = .bob
"from foo.bar import baz as bob" >
= __import__('foo.bar', globals(), locals(), ['baz', 'bob'], -1)
bob = .baz
Thanks for clarifying this! I'd say the current wording of the docs is
non-obvious
in that regard and your examples would be a nice addition to them.
There's a perfectly correct approach documented as part of the
__import__ docs that you referenced in your original email. If
developers decide not to use that approach and use an undocumented hack
instead, they have no right to complain when their code doesn't work
properly.
There's a good reason for the hack -- the documented loop over components
+ getattr is both a bit ugly and inefficient (as is
fromlist=[modname.rsplit(".", 1)[-1]]). The variant proposed by Hrvoje
Niksic:
>>> __import__(modname)
>>> mod = sys.modules[modname]
looks more appealing, but comes with the drawback that sys has to be
imported for
that purpose only.
As none of the variants is really in the spirit of Python's zen and the
use case
"given a string of dotted module names, import the tail module" is
really common,
I'd still say we should go with the flow and provide people what they
need explicitly
(i.e. something in the lines of http://bugs.python.org/issue4438 ) --
and disable
the hack on the same occasion (by the way, is there a reason why both
__import__('foo', fromlist=['invalid']) and __import__('foo',
fromlist=['']) don't raise
an error as of now?). This way the hacks will be eventually fixed in all
those 2000
lines listed in Google Code.
Perhaps 3.1 and 2.7 would be an appropriate chance for that?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __import__ problems
The variant proposed by Hrvoje Niksic: >>> __import__(modname) >>> mod = sys.modules[modname] looks more appealing, but comes with the drawback that sys has to be imported for that purpose only. That is not a real drawback, as "sys" will certainly be present in the system, so the "importing" boils down to a dict lookup and a variable assignment. I meant that you have to import sys only to access sys.modules (i.e. importing sys may not be necessary otherwise). Compare mod = __import__(modname, submodule=True) with import sys __import__(modname) mod = sys.modules[modname] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __import__ problems
Nick Coghlan wrote: As Hrvoje has pointed out, 'sys' is part of the internal interpreter machinery - it's there as soon as the interpreter starts. The import call for it just grabs it out of the module cache and creates a reference to it in the current namespace. I understand that, but Explicit is better than implicit. --> non-explicit import sys and __import__(modname) to access sys.modules[modname] Simple is better than complex. --> three lines for a common use case instead of one Readability counts. --> why is sys imported in this WSGI app (that shouldn't generally touch any of the common sys.argv, sys.stdout etc)?! (500 lines later) aha, to access sys.modules There should be one-- and preferably only one --obvious way to do it. --> should I use the 3-line idiom? should I use the getattr() idiom? Ideally, __import__() should support the only obvious way. In short, the three-line version doesn't convey the intent in an obvious way and does more work than is necessary. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __import__ problems
> If __import__ was replaced with a version with NON compatible interface,
> "import x.y.z" would break.
But it is not. The proposed __import__(name, submodule=True) has
a compatible interface. All tests pass with
http://bugs.python.org/file12136/issue4438.diff .
As for the imp approach, I've alternatively implemented
imp.import_module() that imports tail modules in
http://bugs.python.org/file12147/imp_import_module.diff
(colour diff in http://dpaste.com/hold/94431/).
IMHO the functionality duplication with __import__ is ugly, but
if it is desired that __import__ does not change (even in
backwards-compatible way), so be it.
The signature and behaviour is as follows:
---
import_module(name, globals, level) -> module
Import the tail module, given dotted module hierarchy in 'name'.
'globals' only determines the package context and is not
modified. 'level' specifies whether to use absolute or relative
imports.
>>> from imp import import_module
>>> import_module('foo.bar.baz')
---
But I'm still +1 for adding 'tail'/'submodule'/'tailmodule'
argument to __import__ instead of providing an almost identical
copy in imp.import_module().
Let me know which of the approaches is desired (if any) and I'll
add tests and update docs.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __import__ problems
Brett Cannon wrote: > The old-hands on python-dev know this is where I plug my import > rewrite vaporware. It will be in 3.1, and as part of it there will be > a new API for handling direct imports. Jacob Kaplan-Moss and I have Sounds good. I can finally rest in peace :) . May I suggest that you disable the hack while you are at it as well so that people will be aware of their misdoings? > talked about Django's need for this as PyCon so I am very aware of > needing this API and it will be addressed in the simplest way possible > (heck, the __import__ API might actually become a wrapper around the > simpler API in the end). Even better (the bracketed part). > And since nothing can go into 3.0 anyway, > there is no need to rush into solving this right now. Agreed, I just wanted to get the ball rolling. Let me know if you want me to do some gruntwork (tests, documentation) once the improved implementation is taking shape. --- As for the other comments, I'll try to wrap things up: * I did get the impression from some posts that it was assumed to be somehow "my problem" -- although indeed seeing both 'foo' and 'foo.' when printing sys.modules in a popular web framework I frequently use makes me wince in discomfort, the hack is present in 2000 lines in various projects as seen in the Google Code Search. * runpy.run_module() is not the solution as it runs the module each time it is called and particularly because access to the submodule object is generally needed (again, look around in the Google Code Search results). * changing the signature of __import__ is out of question both because it would break the existing __import__ replacements and would perpetuate the wrong assumption that it should be directly used (instead of the presently missing simpler interface). --- It looks that the __import__(modname) mod = sys.modules[modname] idiom is the clear winner for the import submodule use case. I've updated http://bugs.python.org/issue4457 with proposed additions to current __import__ docs. Once the simpler interface emerges, the docs should be updated again and __import__ use should be discouraged. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] V8, TraceMonkey, SquirrelFish and Python
As most of you know there's constant struggle on the JavaScript front to get even faster performance out of interpreters. V8, TraceMonkey and SquirrelFish have brought novel ideas to interpreter design, wouldn't it make sense to reap the best bits and bring them to Python? Has anyone delved into the designs and considered their applicability to Python? Hoping-to-see-some-V8-and-Python-teams-collaboration-in-Mountain-View-ly yours, Mart Sõmermaa ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] V8, TraceMonkey, SquirrelFish and Python
On Tue, Jan 27, 2009 at 5:04 PM, Jesse Noller wrote: > Hi Mart, > > This is a better discussion for the python-ideas list. That being > said, there was a thread discussing this last year, see: > > http://mail.python.org/pipermail/python-dev/2008-October/083176.html > > -jesse > Indeed, sorry. Incidentally, there is a similar discussion going on just now. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
