Re: [Python-Dev] Security issue with the tracker

[anatoly techtonik]

On Fri, Apr 13, 2012 at 9:53 PM, Éric Araujo  wrote:
> bugs.python.org already sanitizes the ok_message and Ezio already posted a
> patch to the upstream bug tracker, so I don’t see what else we could do.

I am +1 with Glyph that XSS protection in Roundup is an unreliable
hack. Ezio's patch just prolongs the agony - it doesn't make it
better. Code becomes less maintainable. There are two solutions to
that:

1. Use specialized library such as
http://pypi.python.org/pypi/MarkupSafe/ - benefits - easier
maintenance, to get future fixes without waiting until somebody will
have the time to test attacks on Roundup
2. Quote all HTML on server side and use alternative (wiki) markup for
message decorations
3. Do not allow HTML content to be injected through the URL

> Also note that the Firefox extension NoScript blocks the XSS in this case.

NoScripts blocks everything, doesn't it?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Providing a mechanism for PEP 3115 compliant dynamic class creation

[Daniel Urban]

On Tue, Apr 19, 2011 at 16:10, Nick Coghlan  wrote:
> In reviewing a fix for the metaclass calculation in __build_class__
> [1], I realised that PEP 3115 poses a potential problem for the common
> practice of using "type(name, bases, ns)" for dynamic class creation.
>
> Specifically, if one of the base classes has a metaclass with a
> significant __prepare__() method, then the current idiom will do the
> wrong thing (and most likely fail as a result), since "ns" will
> probably be an ordinary dictionary instead of whatever __prepare__()
> would have returned.
>
> Initially I was going to suggest making __build_class__ part of the
> language definition rather than a CPython implementation detail, but
> then I realised that various CPython specific elements in its
> signature made that a bad idea.

Are you referring to the first 'func' argument? (Which is basically
the body of the "class" statement, if I'm not mistaken).

> Instead, I'm thinking along the lines of an
> "operator.prepare(metaclass, bases)" function that does the metaclass
> calculation dance, invoking __prepare__() and returning the result if
> it exists, otherwise returning an ordinary dict. Under the hood we
> would refactor this so that operator.prepare and __build_class__ were
> using a shared implementation of the functionality at the C level - it
> may even be advisable to expose that implementation via the C API as
> PyType_PrepareNamespace().

__prepare__ also needs the name and optional keyword arguments.  So it
probably should be something like "operator.prepare(name, bases,
metaclass, **kw)". But this way it would need almost the same
arguments as __build_class__(func, name, *bases, metaclass=None,
**kwds).

> The correct idiom for dynamic type creation in a PEP 3115 world would then be:
>
>    from operator import prepare
>    cls = type(name, bases, prepare(type, bases))
>
> Thoughts?

When creating a dynamic type, we may want to do it with a non-empty
namespace. Maybe like this (with the extra arguments mentioned above):

   from operator import prepare
   ns = prepare(name, bases, type, **kwargs)
   ns.update(my_ns)  # add the attributes we want
   cls = type(name, bases, ns)

What about an "operator.build_class(name, bases, ns, **kw)" function?
It would work like this:

   def build_class(name, bases, ns, **kw):
   metaclass = kw.pop('metaclass', type)
   pns = prepare(name, bases, metaclass, **kw)
   pns.update(ns)
   return metaclass(name, bases, pns)

(Where 'prepare' is the same as above).
This way we wouldn't even need to make 'prepare' public, and the new
way to create a dynamic type would be:

   from operator import build_class
   cls = build_class(name, bases, ns, **my_kwargs)


Daniel
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Nick Coghlan]

On Sun, Apr 15, 2012 at 12:59 PM, Guido van Rossum  wrote:
> Hm... Can you give an example of a library that needs a real file?
> That sounds like a poorly designed API.

If you're invoking a separate utility (e.g. via it's command line
interface), you may need a real filesystem path that you can pass
along.

>> The get_file() feature has a neat benefit. Since it transparently
>> extracts files from the loader, users can ship binary extensions and
>> shared libraries (dlls) in a ZIP file and use them without too much hassle.
>
> Yeah, DLLs are about the only example I can think of where even a
> virtual filesystem doesn't help...

An important example, though. However, I still don't believe it is
something we should necessarily be rushing into implementing in the
standard library in the *same* release that finally completes the
conversion started so long ago with PEP 302.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making the import machinery explicit

[Nick Coghlan]

Hooray for finally having this to the point where it has been pushed to trunk :)

On Sun, Apr 15, 2012 at 8:16 AM, Brett Cannon  wrote:
> Once again, it's just code that needs updating to run on Python 3.3 so I
> don't view it as a concern. Going from list.append() to list.insert() (even
> if its ``list.insert(hook, len(list)-2)``) is not exactly difficult.

I'm not sure you can so blithely wave away the "check this before the
standard hooks" problem. If the recommended approach becomes to insert
new hooks at the *start* of path_hooks and meta_path, then that should
work fairly well, since the new additions will take precedence
regardless of what other changes have already been made. However,
trying to be clever and say "before the standard hooks, but after
everything else" is fraught with peril, since there may be hooks
present in the lists *after* the standard ones so naive counting
wouldn't work.

As far as the guidelines for managing the import state go, it may be
worth having public "importlib.default_path_hooks" and
"importlib.default_meta_path" attributes.

Then "clearing" the hooks would just be a matter of resetting them
back to defaults: "sys.path_hooks[:] = importlib.default_path_hooks".
You could also locate them in the hooks list correctly by checking
"for i, hook in enumerate(sys.path_hooks): if hook is
importlib.default_path_hooks[0]: break"

Alternatively, it may be simpler to just expose a less granular
"reset_import_hooks()" function that restores meta_path and path_hooks
back to their default state (the defaults could then be private
attributes rather than public ones) and invalidates all the caches.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Providing a mechanism for PEP 3115 compliant dynamic class creation

[Nick Coghlan]

/me pages thoughts from 12 months ago back into brain...

On Sun, Apr 15, 2012 at 7:36 PM, Daniel Urban  wrote:
> On Tue, Apr 19, 2011 at 16:10, Nick Coghlan  wrote:
>> Initially I was going to suggest making __build_class__ part of the
>> language definition rather than a CPython implementation detail, but
>> then I realised that various CPython specific elements in its
>> signature made that a bad idea.
>
> Are you referring to the first 'func' argument? (Which is basically
> the body of the "class" statement, if I'm not mistaken).

Yup, I believe that was my main objection to exposing __build_class__
directly. There's no obligation for implementations to build a
throwaway function to evaluate a class body.

> __prepare__ also needs the name and optional keyword arguments.  So it
> probably should be something like "operator.prepare(name, bases,
> metaclass, **kw)". But this way it would need almost the same
> arguments as __build_class__(func, name, *bases, metaclass=None,
> **kwds).

True.

>> The correct idiom for dynamic type creation in a PEP 3115 world would then 
>> be:
>>
>>    from operator import prepare
>>    cls = type(name, bases, prepare(type, bases))
>>
>> Thoughts?
>
> When creating a dynamic type, we may want to do it with a non-empty
> namespace. Maybe like this (with the extra arguments mentioned above):
>
>   from operator import prepare
>   ns = prepare(name, bases, type, **kwargs)
>   ns.update(my_ns)  # add the attributes we want
>   cls = type(name, bases, ns)
>
> What about an "operator.build_class(name, bases, ns, **kw)" function?
> It would work like this:
>
>   def build_class(name, bases, ns, **kw):
>       metaclass = kw.pop('metaclass', type)
>       pns = prepare(name, bases, metaclass, **kw)
>       pns.update(ns)
>       return metaclass(name, bases, pns)
>
> (Where 'prepare' is the same as above).
> This way we wouldn't even need to make 'prepare' public, and the new
> way to create a dynamic type would be:
>
>   from operator import build_class
>   cls = build_class(name, bases, ns, **my_kwargs)

No, I think we would want to expose the created namespace directly -
that way people can use update(), direct assigment, exec(), eval(), or
whatever other mechanism they choose to handle the task of populating
the namespace. However, a potentially cleaner way to do that might be
offer use an optional callback API rather than exposing a separate
public prepare() function. Something like:

def build_class(name, bases=(), kwds=None, eval_body=None):
metaclass, ns = _prepare(name, bases, kwds)
if eval_body is not None:
eval_body(ns)
return metaclass(name, bases, ns)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Nick Coghlan]

On Sun, Apr 15, 2012 at 8:32 AM, Guido van Rossum  wrote:
> Funny, I was just thinking about having a simple standard API that
> will let you open files (and list directories) relative to a given
> module or package regardless of how the thing is loaded. If we
> guarantee that there's always a __loader__ that's a first step, though
> I think we may need to do a little more to get people who currently do
> things like open(os.path.join(os.path.basename(__file__),
> 'some_file_name') to switch. I was thinking of having a stdlib
> function that you give a module/package object, a relative filename,
> and optionally a mode ('b' or 't') and returns a stream -- and sibling
> functions that return a string or bytes object (depending on what API
> the user is using either the stream or the data can be more useful).
> What would we call thos functions and where would the live?

We already offer pkgutil.get_data() for the latter API:
http://docs.python.org/library/pkgutil#pkgutil.get_data

There's no get_file() or get_filename() equivalent, since there's no
relevant API formally defined for PEP 302 loader objects (the closest
we have is get_filename(), which is only defined for the actual module
objects, not for arbitrary colocated files).

Now that importlib is the official import implementation, and is fully
PEP 302 compliant, large sections of pkgutil should either be
deprecated (the import emulation) or updated to be thin wrappers
around importlib (the package walking components and other utility
functions).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rebuild importlib.h to incorporate added comments.

[Antoine Pitrou]

On Sun, 15 Apr 2012 03:50:06 +0200
brett.cannon  wrote:
> http://hg.python.org/cpython/rev/6a77697d2a63
> changeset:   76311:6a77697d2a63
> user:Brett Cannon 
> date:Sat Apr 14 21:18:48 2012 -0400
> summary:
>   Rebuild importlib.h to incorporate added comments.

Isn't there a Makefile rule to rebuild it automatically?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rebuild importlib.h to incorporate added comments.

[Georg Brandl]

On 15.04.2012 14:53, Antoine Pitrou wrote:

On Sun, 15 Apr 2012 03:50:06 +0200
brett.cannon  wrote:

 http://hg.python.org/cpython/rev/6a77697d2a63
 changeset:   76311:6a77697d2a63
 user:Brett Cannon
 date:Sat Apr 14 21:18:48 2012 -0400
 summary:
   Rebuild importlib.h to incorporate added comments.


Isn't there a Makefile rule to rebuild it automatically?


See the "importlib is now bootstrapped" thread for some problems with that.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions

[Victor Stinner]

> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/

FYI there is no time.thread_time() function. It would only be
available on Windows and Linux. It does not use seconds but CPU
cycles. No module or program of the Python source code need such
function, whereas all other functions added by the PEP already have
users in the Python source code, see the Rationale section. For Linux,
CLOCK_THREAD_CPUTIME_ID is already available in Python 3.3. For
Windows, you can get GetThreadTimes() using ctypes or win32process.

> time.process_time()
> ^^^
>
> Pseudo-code [#pseudo]_::
>
>    if os.name == 'nt':
>        def process_time():
>            handle = win32process.GetCurrentProcess()
>            process_times = win32process.GetProcessTimes(handle)
>            return (process_times['UserTime'] +
> process_times['KernelTime']) * 1e-7
>    else:
>        import os
>        ...
>
>        def process_time():
>            ...
>            return _time.clock()

Is the C clock() function available on all platforms? timemodule.c
checks for HAVE_CLOCK, but test_time checks that time.clock() is
defined and does not fail since the changeset 4de05cbf5ad1, Dec 06
1996. If clock() is not available on all platforms,
time.process_time() documentation should be fixed.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions

[M.-A. Lemburg]

Victor Stinner wrote:
> Hi,
> 
> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/
> 
> The implementation of the PEP can be found in this issue:
> http://bugs.python.org/issue14428
> 
> I post a simplified version for readability and to focus on changes
> introduced by the PEP. Removed sections: Existing Functions,
> Deprecated Function, Glossary, Hardware clocks, Operating system time
> functions, System Standby, Links.

Looks good.

I'd suggest to also include a tool or API to determine the
real resolution of a time function (as opposed to the advertised
one). See pybench's clockres.py helper as example. You often
find large differences between the advertised resolution and
the available one, e.g. while process timers often advertise
very good resolution, they are in fact often only updated
at very coarse rates.

E.g. compare the results of clockres.py on Linux:

Clock resolution of various timer implementations:
time.clock:1.000us
time.time: 0.954us
systimes.processtime:999.000us

and FreeBSD:

Clock resolution of various timer implementations:
time.clock: 7812.500us
time.time: 1.907us
systimes.processtime:  1.000us

and Mac OS X:

Clock resolution of various timer implementations:
time.clock:1.000us
time.time: 0.954us
systimes.processtime:  1.000us

Regarding changing pybench:
pybench has to stay backwards incompatible with
earlier releases to make it possible to compare timings.
You can add support for new timers to pybench, but please leave
the existing timers and defaults in place.

> ---
> 
> PEP: 418
> Title: Add monotonic time, performance counter and process time functions
> Version: f2bb3f74298a
> Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012)
> Author: Cameron Simpson , Jim Jewett
> , Victor Stinner 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 26-March-2012
> Python-Version: 3.3
> 
> Abstract
> 
> 
> This PEP proposes to add ``time.get_clock_info(name)``,
> ``time.monotonic()``, ``time.perf_counter()`` and
> ``time.process_time()`` functions to Python 3.3.
> 
> Rationale
> =
> 
> If a program uses the system time to schedule events or to implement
> a timeout, it will not run events at the right moment or stop the
> timeout too early or too late when the system time is set manually or
> adjusted automatically by NTP.  A monotonic clock should be used
> instead to not be affected by system time updates:
> ``time.monotonic()``.
> 
> To measure the performance of a function, ``time.clock()`` can be used
> but it is very different on Windows and on Unix.  On Windows,
> ``time.clock()`` includes time elapsed during sleep, whereas it does
> not on Unix.  ``time.clock()`` precision is very good on Windows, but
> very bad on Unix.  The new ``time.perf_counter()`` function should be
> used instead to always get the most precise performance counter with a
> portable behaviour (ex: include time spend during sleep).
> 
> To measure CPU time, Python does not provide directly a portable
> function.  ``time.clock()`` can be used on Unix, but it has a bad
> precision.  ``resource.getrusage()`` can also be used on Unix, but it
> requires to get fields of a structure and compute the sum of time
> spent in kernel space and user space.  The new ``time.process_time()``
> function acts as a portable counter that always measures CPU time
> (doesn't include time elapsed during sleep) and has the best available
> precision.
> 
> Each operating system implements clocks and performance counters
> differently, and it is useful to know exactly which function is used
> and some properties of the clock like its resolution and its
> precision.  The new ``time.get_clock_info()`` function gives access to
> all available information of each Python time function.
> 
> New functions:
> 
> * ``time.monotonic()``: timeout and scheduling, not affected by system
>   clock updates
> * ``time.perf_counter()``: benchmarking, most precise clock for short
>   period
> * ``time.process_time()``: profiling, CPU time of the process
> 
> Users of new functions:
> 
> * time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
>   telnet and threading modules to implement timeout
> * time.perf_counter(): trace and timeit modules, pybench program
> * time.process_time(): profile module
> * time.get_clock_info(): pybench program to display information about the
>   timer like the precision or the resolution
> 
> The ``time.clock()`` function is deprecated because it is not
> portable: it behaves differently depending on the operating system.
> ``time.perf_counter()`` or ``time.process_time()`` should be used
> instead, depending on your requirements. ``time.clock()`` is marked as
> deprecated but is not planned for removal.
> 
> 
> Python functions
> 
> 
>

Re: [Python-Dev] making the import machinery explicit

[Brett Cannon]

On Sun, Apr 15, 2012 at 07:26, Nick Coghlan  wrote:

> Hooray for finally having this to the point where it has been pushed to
> trunk :)
>
> On Sun, Apr 15, 2012 at 8:16 AM, Brett Cannon  wrote:
> > Once again, it's just code that needs updating to run on Python 3.3 so I
> > don't view it as a concern. Going from list.append() to list.insert()
> (even
> > if its ``list.insert(hook, len(list)-2)``) is not exactly difficult.
>
> I'm not sure you can so blithely wave away the "check this before the
> standard hooks" problem. If the recommended approach becomes to insert
> new hooks at the *start* of path_hooks and meta_path, then that should
> work fairly well, since the new additions will take precedence
> regardless of what other changes have already been made. However,
> trying to be clever and say "before the standard hooks, but after
> everything else" is fraught with peril, since there may be hooks
> present in the lists *after* the standard ones so naive counting
> wouldn't work.
>

Well, I personally say always insert to the front of sys.meta_path because
getting precedence right is always difficult when you are trying to insert
into the middle of a list. I mean this issue could happen in any
application that have multiple meta path finders and their is an explicit
order that is desired that does not simply fall through from the way code
is called.


>
> As far as the guidelines for managing the import state go, it may be
> worth having public "importlib.default_path_hooks" and
> "importlib.default_meta_path" attributes.
>

I don't think it is. ``importlib.default_meta_path =
[importlib.machinery.PathFinder]`` and ``importlib.default_path_hooks =
[importlib.machinery.some_name_I_have_not_chosen_yet,
zipimport.whatever_its_called]`` is not exactly complicated and if people
are not reading the docs closely enough to realize that is what the
defaults are they are already asking for trouble when mucking around with
import.


>
> Then "clearing" the hooks would just be a matter of resetting them
> back to defaults: "sys.path_hooks[:] = importlib.default_path_hooks".
> You could also locate them in the hooks list correctly by checking
> "for i, hook in enumerate(sys.path_hooks): if hook is
> importlib.default_path_hooks[0]: break"
>

You do realize people will forget the [:] and end up simply screwing up
that original list, right? =)


>
> Alternatively, it may be simpler to just expose a less granular
> "reset_import_hooks()" function that restores meta_path and path_hooks
> back to their default state (the defaults could then be private
> attributes rather than public ones) and invalidates all the caches.
>

What about sys.path_importer_cache: all of it or just NullImporter/None
entries (or should that be a boolean to this function)? And shouldn't it be
called reset_import() with the level of changes you are proposing the
function make?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Suggested addition to PEP 8 for context managers

[Raymond Hettinger]

We should publish some advice on creating content managers.

Context managers are a general purpose tool but have a primary
use case of creating and releasing resources.  This creates an
expectation that that is what the context managers are doing unless
they explicitly say otherwise.

For example in the following calls, the content managers are responsible
for acquiring and releasing a resource.  This is a good and clean design:

with open(filename) as f: ...  # will release the file resource when 
done

with lock: # will acquire and release 
the lock.

However, in the case that someone wants to create a context manager that does
something other than acquiring and releasing resources, then they should
create a separate content manager so that the behavior will be named.

In other words, if the behavior is going to be the common and expected case,
it is okay to add __enter__ and __exit__ to existing classes (as was done
for locks and files).  However, if the behavior is going to do something else,
then the __enter__ and __exit__ methods need to be in a new class or
factory function.

For example, given the typical uses of context managers, I would expect the 
following code to automatically close the database connection:

 with sqlite3.connect(filename) as db:
  ...

Instead, the context manager implements a different behavior.  It would
have been better if that behavior had been given a name:

db = sqlite3.connect(filename)
with auto_commit_or_rollback(db):
  # do a transaction

Explicit beats implicit whenever the implicit behavior would deviate from 
expected norms.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Rebuild importlib.h to incorporate added comments.

[Brett Cannon]

On Sun, Apr 15, 2012 at 10:42, Georg Brandl  wrote:

> On 15.04.2012 14:53, Antoine Pitrou wrote:
>
>> On Sun, 15 Apr 2012 03:50:06 +0200
>> brett.cannon>
>>  wrote:
>>
>>>  
>>> http://hg.python.org/cpython/**rev/6a77697d2a63
>>>  changeset:   76311:6a77697d2a63
>>>  user:Brett Cannon
>>>  date:Sat Apr 14 21:18:48 2012 -0400
>>>  summary:
>>>   Rebuild importlib.h to incorporate added comments.
>>>
>>
>> Isn't there a Makefile rule to rebuild it automatically?
>>
>
> See the "importlib is now bootstrapped" thread for some problems with that.
>
>
In this instance it was because I tested using importlib directly instead
of using import, so I just forgot to build before committing.


>  Georg
>
>
> __**_
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions

[Victor Stinner]

2012/4/15 M.-A. Lemburg :
> I'd suggest to also include a tool or API to determine the
> real resolution of a time function (as opposed to the advertised
> one). See pybench's clockres.py helper as example.

The PEP includes such tool, but I forgot to mention it in the PEP:
http://hg.python.org/peps/file/tip/pep-0418/clock_precision.py

It is based on clockres.py from pybench. I used this tools to fill the
"Precision in Python" column of the different tables. The "Precision"
is the precision announced by the OS, whereas the "Precision in
Python" is the effictive precision in Python.

The full PEP includes results of different benchmarks: performance of
hardware clocks and performance of the different OS time functions.

> E.g. compare the results of clockres.py on Linux:
>
> Clock resolution of various timer implementations:
> time.clock:            1.000us
> time.time:                 0.954us
> systimes.processtime:    999.000us
>
> and FreeBSD:
>
> Clock resolution of various timer implementations:
> time.clock:             7812.500us
> time.time:                 1.907us
> systimes.processtime:      1.000us

Cool, I found similar numbers :-)

> and Mac OS X:
>
> Clock resolution of various timer implementations:
> time.clock:                1.000us
> time.time:                 0.954us
> systimes.processtime:      1.000us

I will include these numbers on Mac OS X to the PEP.

> Regarding changing pybench:
> pybench has to stay backwards incompatible with
> earlier releases to make it possible to compare timings.
> You can add support for new timers to pybench, but please leave
> the existing timers and defaults in place.

I suppose that you are talking about this change:

-# Choose platform default timer
-if sys.platform[:3] == 'win':
-# On WinXP this has 2.5ms resolution
-TIMER_PLATFORM_DEFAULT = TIMER_TIME_CLOCK
-else:
-# On Linux this has 1ms resolution
-TIMER_PLATFORM_DEFAULT = TIMER_TIME_TIME
+TIMER_PLATFORM_DEFAULT = TIMER_TIME_PERF_COUNTER

from http://bugs.python.org/file25202/perf_counter_process_time.patch

It does not change the OS clock on Windows, only on Unix:
CLOCK_REALTIME (gettimeofday() for Python 3.2 and earlier) is replaced
with CLOCK_MONOTONIC. This change should only give a different result
if the system time is changed during the benchmark.

I'm ok to keep the default timer if you consider the change as incompatible.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Providing a mechanism for PEP 3115 compliant dynamic class creation

[Daniel Urban]

On Sun, Apr 15, 2012 at 13:48, Nick Coghlan  wrote:
> /me pages thoughts from 12 months ago back into brain...

Sorry about that, I planned to do this earlier...

> On Sun, Apr 15, 2012 at 7:36 PM, Daniel Urban  wrote:
>> On Tue, Apr 19, 2011 at 16:10, Nick Coghlan  wrote:
>>> Initially I was going to suggest making __build_class__ part of the
>>> language definition rather than a CPython implementation detail, but
>>> then I realised that various CPython specific elements in its
>>> signature made that a bad idea.
>>
>> Are you referring to the first 'func' argument? (Which is basically
>> the body of the "class" statement, if I'm not mistaken).
>
> Yup, I believe that was my main objection to exposing __build_class__
> directly. There's no obligation for implementations to build a
> throwaway function to evaluate a class body.
>
>> __prepare__ also needs the name and optional keyword arguments.  So it
>> probably should be something like "operator.prepare(name, bases,
>> metaclass, **kw)". But this way it would need almost the same
>> arguments as __build_class__(func, name, *bases, metaclass=None,
>> **kwds).
>
> True.
>
>>> The correct idiom for dynamic type creation in a PEP 3115 world would then 
>>> be:
>>>
>>>    from operator import prepare
>>>    cls = type(name, bases, prepare(type, bases))
>>>
>>> Thoughts?
>>
>> When creating a dynamic type, we may want to do it with a non-empty
>> namespace. Maybe like this (with the extra arguments mentioned above):
>>
>>   from operator import prepare
>>   ns = prepare(name, bases, type, **kwargs)
>>   ns.update(my_ns)  # add the attributes we want
>>   cls = type(name, bases, ns)
>>
>> What about an "operator.build_class(name, bases, ns, **kw)" function?
>> It would work like this:
>>
>>   def build_class(name, bases, ns, **kw):
>>       metaclass = kw.pop('metaclass', type)
>>       pns = prepare(name, bases, metaclass, **kw)
>>       pns.update(ns)
>>       return metaclass(name, bases, pns)
>>
>> (Where 'prepare' is the same as above).
>> This way we wouldn't even need to make 'prepare' public, and the new
>> way to create a dynamic type would be:
>>
>>   from operator import build_class
>>   cls = build_class(name, bases, ns, **my_kwargs)
>
> No, I think we would want to expose the created namespace directly -
> that way people can use update(), direct assigment, exec(), eval(), or
> whatever other mechanism they choose to handle the task of populating
> the namespace. However, a potentially cleaner way to do that might be
> offer use an optional callback API rather than exposing a separate
> public prepare() function. Something like:
>
>    def build_class(name, bases=(), kwds=None, eval_body=None):
>        metaclass, ns = _prepare(name, bases, kwds)
>        if eval_body is not None:
>            eval_body(ns)
>        return metaclass(name, bases, ns)

That seems more flexible indeed. I will try to make a patch next week,
if that's OK.


Daniel
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Glyph]

On Apr 14, 2012, at 7:59 PM, Guido van Rossum wrote:

> On Sat, Apr 14, 2012 at 5:06 PM, Christian Heimes  wrote:
>> Am 15.04.2012 00:56, schrieb Guido van Rossum:
>>> Well, if it's a real file, and you need a stream, that's efficient,
>>> and if you need the data, you can read it. But if it comes from a
>>> loader, and you need a stream, you'd have to wrap it in a StringIO
>>> instance. So having two APIs, one to get a stream, and one to get the
>>> data, allows the implementation to be more optimal -- it would be bad
>>> to wrap a StringIO instance around data only so you can read the data
>>> from the stream again...
>> 
>> We need a third way to access a file. The two methods get_data() and
>> get_stream() aren't sufficient for libraries that need a read file that
>> lives on the file system. In order to have real files the loader (or
>> some other abstraction layer) needs to create a temporary directory for
>> the current process and clean it up when the process ends. The file is
>> saved to the temporary directory the first time it's accessed.
> 
> Hm... Can you give an example of a library that needs a real file?
> That sounds like a poorly designed API.

Lots of C libraries use filenames or FILE*s where they _should_ be using some 
much more abstract things; i.e., constellations of function pointers that are 
isomorphic to Python's "file-like objects".  Are these APIs poorly designed?  
Sure, but they also exist ;).

> Perhaps you're talking about APIs that take a filename instead of a
> stream? Maybe for those it would be best to start getting serious
> about a virtual filesystem... (Sorry, probably python-ideas stuff).

twisted.python.filepath... ;-)

>> The get_file() feature has a neat benefit. Since it transparently
>> extracts files from the loader, users can ship binary extensions and
>> shared libraries (dlls) in a ZIP file and use them without too much hassle.
> 
> Yeah, DLLs are about the only example I can think of where even a
> virtual filesystem doesn't help...

In a previous life, I was frequently exposed to proprietary game-engine things 
that could only load resources (3D models, audio files, textures) from actual 
real files, and I had to do lots of unpacking stuff either from things tacked 
on to a .exe or inside a zip file.  (I don't know how common this is any more 
in that world but I suspect "very".)

Unfortunately all the examples I can think of off the top of my head were in 
proprietary, now defunct code; but this is exactly the sort of polish that 
open-sourcing tends to apply, so I would guess problematic code in this regard 
would more often be invisible.

-glyph
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Glyph]

On Apr 14, 2012, at 3:32 PM, Guido van Rossum wrote:

> Funny, I was just thinking about having a simple standard API that
> will let you open files (and list directories) relative to a given
> module or package regardless of how the thing is loaded.


Twisted has such a thing, mostly written by me, called twisted.python.modules.

Sorry if I'm repeating myself here, I know I've brought it up on this list 
before, but it seems germane to this thread.  I'd be interested in getting 
feedback from the import-wizards participating in this thread in case it is 
doing anything bad (in particular I'd like to make sure it will keep working in 
future versions of Python), but I think it may provide quite a good template 
for a standard API.

The code's here: 


The API is fairly simple.

>>> from twisted.python.modules import getModule
>>> e = getModule("email") # get an abstract "module" object (un-loaded)
>>> e
PythonModule<'email'>
>>> walker = e.walkModules() # walk the module hierarchy
>>> walker.next()
PythonModule<'email'>
>>> walker.next()
PythonModule<'email._parseaddr'>
>>> walker.next() # et cetera
PythonModule<'email.base64mime'>
>>> charset = e["charset"] # get the 'charset' child module of the 'e' package
>>> charset.filePath
FilePath('.../lib/python2.7/email/charset.py')
>>> charset.filePath.parent().children() # list the directory containing 
>>> charset.py

Worth pointing out is that although in this example it's a FilePath, it could 
also be a ZipPath if you imported stuff from a zipfile.  We have an adapter 
that inspects path_importer_cache and produces appropriately-shaped 
filesystem-like objects depending on where your module was imported from.  
Thank you to authors of PEP 302; that was my religion while writing this code.

You can also, of course, ask to load something once you've identified it with 
the traversal API:

>>> charset.load()


You can also ask questions like this, which are very useful when debugging 
setup problems:

>>> ifaces = getModule("twisted.internet.interfaces")
>>> ifaces.pathEntry
PathEntry
>>> list(ifaces.pathEntry.iterModules())
[PythonModule<'setup'>, PythonModule<'twisted'>]

This asks what sys.path entry is responsible twisted.internet.interfaces, and 
then what other modules could be loaded from there.  Just 'setup' and 'twisted' 
indicates that this is a development install (not surprising for one of my 
computers), since site-packages would be much more crowded.

The idiom for saying "there's a file installed near this module, and I'd like 
to grab it as a string", is pretty straightforward:

from twisted.python.modules import getModule
mod = getModule(__name__).filePath.sibling("my-file").open().read()

And hopefully it's obvious from this idiom how one might get the pathname, or a 
stream rather than the bytes.

-glyph___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Update importlib.h

[Antoine Pitrou]

On Sun, 15 Apr 2012 23:56:18 +0200
brett.cannon  wrote:
> http://hg.python.org/cpython/rev/096653de404d
> changeset:   76332:096653de404d
> user:Brett Cannon 
> date:Sun Apr 15 17:47:19 2012 -0400
> summary:
>   Update importlib.h

I wonder if we could somehow set importlib.h as binary so that
Mercurial doesn't give us huge diffs each time the Python source is
modified.
Adding a NUL byte in the generated file would probably be sufficient.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Edit the rejected PEP 416 (frozendict) to mention the newly added types.MappingProxyType

[Victor Stinner]

Hi,

The frozendict (PEP 416) was rejected, but I just added the
types.MappingProxyType. This type is not new, it existed since Python
2.2 as the internal dict_proxy type. See also the issue #14386.

I would like to know if I can edit the rejected PEP, or if Guido
prefers to do it, to mention the new type? The "Rejection Notice"
section currently ends with "On the other hand, exposing the existing
read-only dict proxy as a built-in type sounds good to me. (It would
need to be changed to allow calling the constructor.) GvR."

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Barry Warsaw]

On Apr 14, 2012, at 03:32 PM, Guido van Rossum wrote:

>Funny, I was just thinking about having a simple standard API that
>will let you open files (and list directories) relative to a given
>module or package regardless of how the thing is loaded.

I tend to use the "basic resource access" API of pkg_resources.

http://peak.telecommunity.com/DevCenter/PkgResources#basic-resource-access

I'm not suggesting that we adopt all of pkg_resources, but I think the 5
functions listed there, plus resource_filename() (from the next section)
provide basic functionality I've found very useful.

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Barry Warsaw]

On Apr 15, 2012, at 02:12 PM, Glyph wrote:

>Twisted has such a thing, mostly written by me, called
>twisted.python.modules.
>
>Sorry if I'm repeating myself here, I know I've brought it up on this list
>before, but it seems germane to this thread.  I'd be interested in getting
>feedback from the import-wizards participating in this thread in case it is
>doing anything bad (in particular I'd like to make sure it will keep working
>in future versions of Python), but I think it may provide quite a good
>template for a standard API.
>
>The code's here: 
>
>
>The API is fairly simple.
>
 from twisted.python.modules import getModule
 e = getModule("email") # get an abstract "module" object (un-loaded)

Got a PEP 8 friendly version? :)

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Edit the rejected PEP 416 (frozendict) to mention the newly added types.MappingProxyType

[Guido van Rossum]

Go ahead and update the PEP!

On Sunday, April 15, 2012, Victor Stinner wrote:

> Hi,
>
> The frozendict (PEP 416) was rejected, but I just added the
> types.MappingProxyType. This type is not new, it existed since Python
> 2.2 as the internal dict_proxy type. See also the issue #14386.
>
> I would like to know if I can edit the rejected PEP, or if Guido
> prefers to do it, to mention the new type? The "Rejection Notice"
> section currently ends with "On the other hand, exposing the existing
> read-only dict proxy as a built-in type sounds good to me. (It would
> need to be changed to allow calling the constructor.) GvR."
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Require loaders set __package__ and __loader__

[Glyph]

On Apr 15, 2012, at 6:38 PM, Barry Warsaw wrote:

> On Apr 15, 2012, at 02:12 PM, Glyph wrote:
> 
>> Twisted has such a thing, mostly written by me, called
>> twisted.python.modules.
>> 
>> Sorry if I'm repeating myself here, I know I've brought it up on this list
>> before, but it seems germane to this thread.  I'd be interested in getting
>> feedback from the import-wizards participating in this thread in case it is
>> doing anything bad (in particular I'd like to make sure it will keep working
>> in future versions of Python), but I think it may provide quite a good
>> template for a standard API.
>> 
>> The code's here: 
>> 
>> 
>> The API is fairly simple.
>> 
> from twisted.python.modules import getModule
> e = getModule("email") # get an abstract "module" object (un-loaded)
> 
> Got a PEP 8 friendly version? :)

No, but I'd be happy to do the translation manually if people actually prefer 
the shape of this API!

I am just pointing it out as a source of inspiration for whatever comes next, 
which I assume will be based on pkg_resources.

-glyph
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions

[Victor Stinner]

> time.perf_counter()
> ^^^
>
> Performance counter with the highest available precision to measure a
> duration.  It does include time elapsed during sleep and is
> system-wide.  The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid and is a number of seconds.

It's maybe time for bikeshedding! Glyph wrote me in private:
"IMHO, perf_counter should be performance_counter() or
high_precision(); the abbreviation is silly :)"

The time module has other abbreviated names. I don't have a preference
between time.perf_counter() or time.performance_counter().

Solaris provides CLOCK_HIGHRES, "the nonadjustable, high-resolution
clock." If we map CLOCK_xxx names to functions name, we have:

 * CLOCK_MONOTONIC: time.monotonic()
 * CLOCK_HIGHRES: time.highres()

(whereas Windows provides QueryPerformanceCounter -> performance_counter)

I suppose that most people don't care that "resolution" and
"precision" are different things.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making the import machinery explicit

[Nick Coghlan]

On Mon, Apr 16, 2012 at 2:31 AM, Brett Cannon  wrote:
> What about sys.path_importer_cache: all of it or just NullImporter/None
> entries (or should that be a boolean to this function)? And shouldn't it be
> called reset_import() with the level of changes you are proposing the
> function make?

Hmm, perhaps the simpler suggestion is: "If you want a clean import
state, use multiprocessing or the subprocess module to invoke a new
instance of python" :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Providing a mechanism for PEP 3115 compliant dynamic class creation

[Nick Coghlan]

On Mon, Apr 16, 2012 at 5:34 AM, Daniel Urban  wrote:
> On Sun, Apr 15, 2012 at 13:48, Nick Coghlan  wrote:
>> /me pages thoughts from 12 months ago back into brain...
>
> Sorry about that, I planned to do this earlier...

No worries - good to have someone following up on it, since it had
completely dropped off my own radar :)

>> No, I think we would want to expose the created namespace directly -
>> that way people can use update(), direct assigment, exec(), eval(), or
>> whatever other mechanism they choose to handle the task of populating
>> the namespace. However, a potentially cleaner way to do that might be
>> offer use an optional callback API rather than exposing a separate
>> public prepare() function. Something like:
>>
>>    def build_class(name, bases=(), kwds=None, eval_body=None):
>>        metaclass, ns = _prepare(name, bases, kwds)
>>        if eval_body is not None:
>>            eval_body(ns)
>>        return metaclass(name, bases, ns)
>
> That seems more flexible indeed. I will try to make a patch next week,
> if that's OK.

Sure, just create a new tracker issue and assign it to me. You already
know better than most what the _prepare() step needs to do :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] making the import machinery explicit

[Brett Cannon]

On Sun, Apr 15, 2012 at 22:03, Nick Coghlan  wrote:

> On Mon, Apr 16, 2012 at 2:31 AM, Brett Cannon  wrote:
> > What about sys.path_importer_cache: all of it or just NullImporter/None
> > entries (or should that be a boolean to this function)? And shouldn't it
> be
> > called reset_import() with the level of changes you are proposing the
> > function make?
>
> Hmm, perhaps the simpler suggestion is: "If you want a clean import
> state, use multiprocessing or the subprocess module to invoke a new
> instance of python" :)
>
>
Yeah, kinda. =) This is why testing import (as you know) is such an utter
pain.

-Brett



>  Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggested addition to PEP 8 for context managers

[Chris Angelico]

On Mon, Apr 16, 2012 at 3:13 AM, Raymond Hettinger
 wrote:
> Instead, the context manager implements a different behavior.  It would
> have been better if that behavior had been given a name:
>
>    db = sqlite3.connect(filename)
>    with auto_commit_or_rollback(db):
>          # do a transaction

I agree that it wants a name. If explicitness is the goal, would this
be more suitable?

db = sqlite3.connect(filename)
with db.begin_transaction() as trans:
  # do a transaction

This way, if a database engine supports multiple simultaneous
transactions, the same syntax can be used.

Chris Angelico
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com