Re: [Python-Dev] Python 3 optimizations...

[Stefan Behnel]

stefan brunthaler, 23.07.2010 08:48:

I guess it would be a good idea to quickly outline my inline caching
approach, so that we all have a basic understanding of how it works.


Yes, that certainly makes it easier to discuss.



If we take for instance the BINARY_ADD instruction, the interpreter
evaluates the actual operand types and chooses the matching operation
implementation at runtime, i.e., operands that are unicode strings
will be concatenated via unicode_concatenate, for float operands on
the other hand, the interpreter would end up invoking float_add via
binary_op1. Now, a very efficient way to achieve purely interpretative
inline caching is to quicken the type-generic BINARY_ADD instruction
to a type-dependent FLOAT_ADD instruction (this technique, i.e.,
inline caching via quickening, is the primary contribution of my ECOOP
paper). Hence, I have a very simple code generator, that generates
type-dependent interpreter instructions in a pre-compile step of the
interpreter, and uses runtime type information to quicken/rewrite
instructions.
Aside of the operators, I have implemented this quickening technique
for FOR_ITER, COMPARE_OP and CALL_FUNCTION instructions.


This sounds like wpython (a CPython derivative with a wider set of byte 
code commands) could benefit from it.


Do I understand correctly that you modify the byte code of 
modules/functions at runtime?




I'm absolutely interested, although not for the CPython project but for
Cython. I wonder how you do inline caching in Python if the methods of a
type can be replaced by whatever at runtime. Could you elaborate on that?


Currently, I only provide optimized derivatives for several separate
call targets, i.e., whether a call target is a C function with
varargs, or a Python function/method--this already eliminates a lot of
overhead from invoking call_function.


Ah, yes, that makes good sense. So you basically add an intermediate step 
to calls that provides faster dispatch for known C functions.




Or do you restrict yourself to builtin types?


Currently, my approach provides optimized derivative instructions for
the standard library, e.g., unicode strings, numerical objects,
containers, and iterators.


I'm interested in the code that determines what can be optimised in what 
way. I read that Jython recently received a contribution that provides type 
information for lots of modules and builtins, but having something like 
that for CPython would be cool.




That might be worth it
already, just think of list.append(). We have an optimistic optimisation for
object.append() in Cython that gives us massive speed-ups in loops that
build lists, even if we don't know at compile time that we are dealing with
lists.


Yes, that sounds like a reasonable thing to do. I could provide much
more optimized derivatives based on application profiles, too. Since I
use a simple code generator for generating the derivatives, it would
also be possible to provide end-users with the means to analyze their
apps and generate optimized instruction derivatives matching their
profile.


Such an approach would also be very useful for Cython. Think of a profiler 
that runs a program in CPython and tells you exactly what static type 
annotations to put where in your Python code to make it compile to a fast 
binary with Cython. Or, even better, it could just spit out a .pxd file 
that you drop next to your .py file and that provides the static type 
information for you.


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Ronald Oussoren]

On 22 Jul, 2010, at 15:40, Barry Warsaw wrote:
> 
> 
> Abstract
> 
> 
> PEP 3147 [1]_ described an extension to Python's import machinery that
> improved the sharing of Python source code, by allowing more than one
> byte compilation file (.pyc) to be co-located with each source file.
> 
> This PEP defines an adjunct feature which allows the co-location of
> extension module files (.so) in a similar manner.  This optional,
> build-time feature will enable downstream distributions of Python to
> more easily provide more than one Python major version at a time.

I guess this is not an explicit goal of this PEP, but the structure is very 
close to supporting multiple system architectures at the same time.  I 
regularly develop code that needs to run on Windows, Linux and OSX and it is 
very convenient to do so in a shared directory tree (locally on one machine and 
accessed using remote mounts on the other ones). This works fine for pure 
python code, but I currently have to resort to tricks for extension modules.
> 
> Proposal
> 
> 
> The configure/compilation options chosen at Python interpreter
> build-time will be encoded in the shared library file name for
> extension modules.  This "tag" will appear between the module base
> name and the operation file system extension for shared libraries.
> 
> The following information *MUST* be included in the shared library
> file name:
> 
> * The Python implementation (e.g. cpython, pypy, jython, etc.)
> * The interpreter's major and minor version numbers
> 
> These two fields are separated by a hyphen and no dots are to appear
> between the major and minor version numbers.  E.g. ``cpython-32``.
> 
> Python implementations *MAY* include additional flags in the file name
> tag as appropriate.  For example, on POSIX systems these flags will
> also contribute to the file name:
> 
> * ``--with-pydebug`` (flag: ``d``)
> * ``--with-pymalloc`` (flag: ``m``)
> * ``--with-wide-unicode`` (flag: ``u``)
> 
> By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so
> shared library file names would appear as ``foo.cpython-32m.so``.
> When the other two flags are also enabled, the file names would be
> ``foo.cpython-32dmu.so``.

A way to generically solve my problem is to add the platform name as well, such 
as "foo.cpython-32m-darwin.so" or "foo.cpython-32mu-linux2.so".

Ronald

smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[stefan brunthaler]

> This sounds like wpython (a CPython derivative with a wider set of byte code
> commands) could benefit from it.
>
I am aware of the wpython project of Cesare di Mauro. I change the
instruction format from bytecode to wordcode, too (because it allows
for more efficient instruction decoding). Contrary to his approach,
however, I do not change the instruction encoding to pack in
additional optimizations. (I hope to have put that correctly; I have
seen his slides about a year ago.)


> Do I understand correctly that you modify the byte code of modules/functions
> at runtime?
>
Yes. Quickening is runtime only optimization technique that rewrites
instructions from a generic instruction to an optimized derivative
(orignally for the Java virtual machine). It is completely hidden from
the compiler and has no other dependencies than the interpreter
dispatch routine itself.



> Ah, yes, that makes good sense. So you basically add an intermediate step to
> calls that provides faster dispatch for known C functions.
>
Exactly. I also contemplated to provide optimized derivatives for all
builtin functions, but never implemented it (lack of time). Based on
quantitative analysis of usage frequency one could very well decide
to, e.g., provide an optimized CALL_FUNCTION derivative for the "len"
function.
Another benefit of using my technique is that a compiler could decide
to inline all of the functions of the optimized derivatives (e.g., the
float_add function call inside my FLOAT_ADD interpreter instruction).
Unfortunately, however, gcc currently does not allow for cross-module
inlining (AFAIR). (Preliminary tests with manually changing the
default inlining size for ceval.c resulted in speedups of up to 1.3 on
my machine, so I think inlinling of function bodies for the optimized
derivatives would boost performance noticeably.)


> I'm interested in the code that determines what can be optimised in what
> way. I read that Jython recently received a contribution that provides type
> information for lots of modules and builtins, but having something like that
> for CPython would be cool.
>
Ok. For this year's PPPJ I wanted to submit a paper realizing my
optimization in Jython. Because of bytecode-rewriting tools, the
interpreter could decide at runtime which optimized derivatives to
generate and add rewriting code that supports the changing instruction
set. Either way (static pre-compiling or dynamic bytecode rewriting
that is), I think that Jython and IronPython would greatly benefit
from applying this optimization technique, because their JIT compilers
would inline the function calls with a high likelihood.


> Such an approach would also be very useful for Cython. Think of a profiler
> that runs a program in CPython and tells you exactly what static type
> annotations to put where in your Python code to make it compile to a fast
> binary with Cython. Or, even better, it could just spit out a .pxd file that
> you drop next to your .py file and that provides the static type information
> for you.
>
Hm, I think you could very easily save away the optimized bytecode
sequence for function calls that would allow you to do that (e.g., you
could save something similar to:
   LOAD_FAST
   LOAD_CONST
   LONG_ADD
or
   LOAD_GLOBAL
   CALL_C_ZERO
)


Cheers,
--stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[Cesare Di Mauro]

2010/7/23 Stefan Behnel 

> stefan brunthaler, 23.07.2010 08:48:
>
> If we take for instance the BINARY_ADD instruction, the interpreter
>> evaluates the actual operand types and chooses the matching operation
>> implementation at runtime, i.e., operands that are unicode strings
>> will be concatenated via unicode_concatenate, for float operands on
>> the other hand, the interpreter would end up invoking float_add via
>> binary_op1. Now, a very efficient way to achieve purely interpretative
>> inline caching is to quicken the type-generic BINARY_ADD instruction
>> to a type-dependent FLOAT_ADD instruction (this technique, i.e.,
>> inline caching via quickening, is the primary contribution of my ECOOP
>> paper). Hence, I have a very simple code generator, that generates
>> type-dependent interpreter instructions in a pre-compile step of the
>> interpreter, and uses runtime type information to quicken/rewrite
>> instructions.
>> Aside of the operators, I have implemented this quickening technique
>> for FOR_ITER, COMPARE_OP and CALL_FUNCTION instructions.
>>
>> This sounds like wpython (a CPython derivative with a wider set of byte
> code commands) could benefit from it.
>

WPython 1.1 does it at compile time, if you enable the new "experimental
integer opcodes" flag.

Similar optimizations were introduced with new opcodes for specialized
string interpolation and joins, which are common operations in Python.

It also added a new opcode GET_GENERATOR which internally uses a faster
function call, which is used also by (the modified) BUILD_CLASS for the same
reason (cut some unnecessary checks and code).

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils reverted in py3k

[Ronald Oussoren]

On 22 Jul, 2010, at 14:01, Tarek Ziadé wrote:

> Note that I'll revert Doc/distutils as well, but I need to check first
> with Ronald a few Mac OS X points.

Not just a few point, the revert completely broke universal builds on OSX. I'm 
working on a fix.

Ronald

> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils reverted in py3k

[Ronald Oussoren]

On 23 Jul, 2010, at 10:12, Ronald Oussoren wrote:

> 
> On 22 Jul, 2010, at 14:01, Tarek Ziadé wrote:
> 
>> Note that I'll revert Doc/distutils as well, but I need to check first
>> with Ronald a few Mac OS X points.
> 
> Not just a few point, the revert completely broke universal builds on OSX. 
> I'm working on a fix.

The build seems to work again, my fix is in r83066. There are a couple of test 
failures left, but those don't seem to be related to distutils.

Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils reverted in py3k

[Tarek Ziadé]

2010/7/23 Ronald Oussoren :
>
> On 23 Jul, 2010, at 10:12, Ronald Oussoren wrote:
>
>>
>> On 22 Jul, 2010, at 14:01, Tarek Ziadé wrote:
>>
>>> Note that I'll revert Doc/distutils as well, but I need to check first
>>> with Ronald a few Mac OS X points.
>>
>> Not just a few point, the revert completely broke universal builds on OSX. 
>> I'm working on a fix.

mmm sorry about that... I though you backported everything in 3.1

>
> The build seems to work again, my fix is in r83066. There are a couple of 
> test failures left, but those don't seem to be related to distutils.

Thanks.  The next task zould be to check what 3,2 doc should be kept
before Docs/distutils is reverted

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Distutils reverted in py3k

[Ronald Oussoren]

On 23 Jul, 2010, at 10:47, Tarek Ziadé wrote:

> 2010/7/23 Ronald Oussoren :
>> 
>> On 23 Jul, 2010, at 10:12, Ronald Oussoren wrote:
>> 
>>> 
>>> On 22 Jul, 2010, at 14:01, Tarek Ziadé wrote:
>>> 
 Note that I'll revert Doc/distutils as well, but I need to check first
 with Ronald a few Mac OS X points.
>>> 
>>> Not just a few point, the revert completely broke universal builds on OSX. 
>>> I'm working on a fix.
> 
> mmm sorry about that... I though you backported everything in 3.1

3.1 is probably broken as well due to the recent move to setting PY_CFLAGS 
instead of CFLAGS.  That is next on my list.

Ronald





smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jul 22, 2010, at 03:58 PM, Ronald Oussoren wrote:

>I guess this is not an explicit goal of this PEP, but the structure is
>very close to supporting multiple system architectures at the same
>time.  I regularly develop code that needs to run on Windows, Linux
>and OSX and it is very convenient to do so in a shared directory tree
>(locally on one machine and accessed using remote mounts on the other
>ones). This works fine for pure python code, but I currently have to
>resort to tricks for extension modules.

[...]

>A way to generically solve my problem is to add the platform name as
>well, such as "foo.cpython-32m-darwin.so" or
>"foo.cpython-32mu-linux2.so".

This could certainly be done in the Windows build, but that wouldn't help
bridge the gap among different POSIX systems.  I'd be open to adding the
platform name to the tag, but I'd probably define it as part of the
implementation field, e.g. foo.cpython-linux2-32m.so.  Or maybe start with the
platform name, e.g.  foo.linux2-cpython-32m.  This isn't a strong preference
though.

Thoughts?
-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Summit EuroPython 2010

[Brett Cannon]

On Thu, Jul 22, 2010 at 13:00, Antoine Pitrou  wrote:

> On Thu, 22 Jul 2010 08:51:57 +0100
> Brett Cannon  wrote:
> >
> > That's an option. I just remember Tim bringing up something about that
> > approach that didn't quite work as a complete replacement for __del__.
> >
> > Basically the whole setting a module's globals to None was done before gc
> > came into the language. Now that it's there it seems that it might work
> to
> > simply let gc clean up the module itself.
>
> There is a patch at http://bugs.python.org/issue812369 for GC-based
> module shutdown, but it doesn't actually remove the setting of module
> globals to None. I think further testing and experimentation would be
> required to validate it.
>

Yep, it was agreed that for this to happen someone would have to write the
patch and see what broke.

-Brett


>
> Regards
>
> Antoine.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New regex module for 3.2?

[Hrvoje Niksic]

On 07/22/2010 01:34 PM, Georg Brandl wrote:

Timings (seconds to run the test suite):

re 26.689  26.015  26.008
regex  26.066  25.797  25.865

So, I thought there wasn't a difference in performance for this use case
(which is compiling a lot of regexes and matching most of them only a
few times in comparison).  However, I found that looking at the regex
caching is very important in this case: re._MAXCACHE is by default set to
100, and regex._MAXCACHE to 1024.  When I set re._MAXCACHE to 1024 before
running the test suite, I get times around 18 (!) seconds for re.


This seems to point to re being significantly *faster* than regexp, even 
in matching, and as such may be something the author would want to look 
into.


Nick writes:

> That still fits with the compile/match performance trade-off changes
> between re and regex though.

The performance trade-off should make regex slower with sufficiently 
small compiled regex cache, when a lot of time is wasted on compilation. 
 But as the cache gets larger (and, for fairness, of the same size in 
both implementations), regex should outperform re.  Georg, would you 
care to measure if there is a difference in performance with an even 
larger cache?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 360 has outdated contents

[Brett Cannon]

On Thu, Jul 22, 2010 at 14:48, Antoine Pitrou  wrote:

>
> Hello all,
>
> PEP 360 - “Externally Maintained Packages” seems to have outdated
> contents.
> First of all, I don't think Optik and wsgiref are externally
> maintained anymore (both seem unmaintained by their original authors).
>

Added a note.


> Second, the version numbers mentioned there could be out of date too
> (especially for ElementTree whose stdlib copy has recently been bumped
> by Florent).
>

I ripped out the sync dates since keeping them up is just a pain.


> Third, it seems json should be added to the list, despite the prominent
> warning at the beginning that:
>
> “No new modules are to be added to this PEP. It has been deemed
> dangerous to codify external maintenance of any code checked into
> Python's code repository. Code contributers should expect Python's
> development methodology to be used for any and all code checked into
> Python's code repository.”
>
>
The PEP lists external maintenance that we agreed to. Bob isn't supposed to
be maintaining json outside of the stdlib, so I am not going to add it.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[Nick Coghlan]

On Fri, Jul 23, 2010 at 1:00 AM, David Cournapeau  wrote:
> On Thu, Jul 22, 2010 at 10:08 PM, stefan brunthaler
>  wrote:
>>> Is the source code under an open source non-copyleft license?
>>>
>> I am (unfortunately) not employed or funded by anybody, so I think
>> that I can license/release the code as I see fit.
>
> If you did this work under your PhD program, you may be more
> restricted than you think. You may want to check with your adviser
> first,

This is actually a good point - some universities have (IMO) weird
ideas regarding the ownership of thesis material. Best to get that
squared away before you publish the code online.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Ronald Oussoren]

On 23 Jul, 2010, at 11:02, Barry Warsaw wrote:

> On Jul 22, 2010, at 03:58 PM, Ronald Oussoren wrote:
> 
>> I guess this is not an explicit goal of this PEP, but the structure is
>> very close to supporting multiple system architectures at the same
>> time.  I regularly develop code that needs to run on Windows, Linux
>> and OSX and it is very convenient to do so in a shared directory tree
>> (locally on one machine and accessed using remote mounts on the other
>> ones). This works fine for pure python code, but I currently have to
>> resort to tricks for extension modules.
> 
> [...]
> 
>> A way to generically solve my problem is to add the platform name as
>> well, such as "foo.cpython-32m-darwin.so" or
>> "foo.cpython-32mu-linux2.so".
> 
> This could certainly be done in the Windows build, but that wouldn't help
> bridge the gap among different POSIX systems.

The windows port isn't a problem for this, it uses a different suffix (".pyd") 
than the unix ports.

>  I'd be open to adding the
> platform name to the tag, but I'd probably define it as part of the
> implementation field, e.g. foo.cpython-linux2-32m.so.  Or maybe start with the
> platform name, e.g.  foo.linux2-cpython-32m.  This isn't a strong preference
> though.

I don't have a strong opionion, but placing the platform name at the start
is probably better to be consistent with sysconfig.get_platform().

Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jul 23, 2010, at 11:48 AM, Ronald Oussoren wrote:

>>  I'd be open to adding the
>> platform name to the tag, but I'd probably define it as part of the
>> implementation field, e.g. foo.cpython-linux2-32m.so.  Or maybe
>> start with the platform name, e.g.  foo.linux2-cpython-32m.  This
>> isn't a strong preference though.
>
>I don't have a strong opionion, but placing the platform name at the
>start is probably better to be consistent with
>sysconfig.get_platform().

What about the architecture (i386, amd64)?  With every increase in length I
start to get more concerned.  We could encode the platform and architecture,
but that gets into cryptic territory.  OTOH, would you really co-install i386
and amd64 shared libraries on the same machine?  (hello NFS ;).

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Nick Coghlan]

On Fri, Jul 23, 2010 at 12:40 AM, Barry Warsaw  wrote:
> Python implementations *MAY* include additional flags in the file name
> tag as appropriate.  For example, on POSIX systems these flags will
> also contribute to the file name:
>
>  * ``--with-pydebug`` (flag: ``d``)
>  * ``--with-pymalloc`` (flag: ``m``)
>  * ``--with-wide-unicode`` (flag: ``u``)
>
> By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so
> shared library file names would appear as ``foo.cpython-32m.so``.
> When the other two flags are also enabled, the file names would be
> ``foo.cpython-32dmu.so``.
>
> (This PEP only addresses build issues on POSIX systems that use the
> ``configure`` script.  While Windows or other platform support is not
> explicitly disallowed under this PEP, platform expertise is needed in
> order to evaluate, describe, and implement support on such platforms.)

This leads me to a question: how do these configure options affect the
PEP 384 stable ABI? That PEP is currently silent on the issue, while
PEP 3149 appears to implicitly assume that "abi3" completely specifies
the ABI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382 progress: import hooks

[Brett Cannon]

On Thu, Jul 22, 2010 at 19:19, P.J. Eby  wrote:

> At 01:51 PM 7/22/2010 +0100, Martin v. Löwis wrote:
>
>> At EuroPython, I sat down with Brett and we propose an approach
>> how namespace packages get along with import hooks. I reshuffled
>> the order in which things get done a little bit, and added a
>> section that elaborates on the hooks.
>>
>> Basically, a finder will need to support a find_path method,
>> return all .pth files, and a loader will need to support a
>> load_module_with_path method, to initialize __path__.
>>
>> Please comment if you think that this needs further changes;
>>
>
> I'm not certain I understand it precisely.  There seem to be some
> ambiguities in the spec, e.g.:
>
> "If fullname is not found, is not a package, or does not have any *.pth
> files, None must be returned."
>
> What does "is not a package" actually mean in that context?


The module is a module but not a package.


>  What happens if an empty list is returned - does that mean the importer is
> saying, "this is a package, whether it has an __init__.py or not?"
>




> As for the "list of strings" returned, is each string the entire contents
> of the .pth file?  Is it to be \n-separated, or is any
> universal-newlines-compatible string accepted?  Is there a particular order
> in which .pth file contents are to be returned?
>
> Regarding load_module_with_path(), how does its specification differ from
> simply creating a module in sys.modules, setting its __path__, and then
> invoking the standard load_module()?  (i.e., is this method actually needed,
> since a correct PEP 302 loader *must* reuse an existing module object in
> sys.modules)


It must reuse the module itself but a proper reload would reset __path__ as
leaving it unchanged is not a proper resetting of the module object. So this
module is needed in order to force the loader


>
>
>
>  I'll hope to start implementing it soon.
>>
>
> Am I correct in understanding that, as written, one would have to redefine
> __import__ to implement this in a library for older Python versions?  Or is
> it implementable as a meta_path importer?
>

Redefine __import__ (unless Martin and I are missing something, but I tried
to think of how to implement this use sys.meta_path and couldn't come up
with a solution).

-Brett


>
>
>  Regards,
>> Martin
>>
>
> Thanks for your work on this, I was just thinking about pinging to see how
> it was going.  ;-)
>
> (I want setuptools 0.7 to be able to supply an add-on module for supporting
> this PEP in older Pythons, so that its current .pth hacks for implementing
> namespace packages can be dropped.)
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Ronald Oussoren]

On 23 Jul, 2010, at 11:54, Barry Warsaw wrote:

> On Jul 23, 2010, at 11:48 AM, Ronald Oussoren wrote:
> 
>>> I'd be open to adding the
>>> platform name to the tag, but I'd probably define it as part of the
>>> implementation field, e.g. foo.cpython-linux2-32m.so.  Or maybe
>>> start with the platform name, e.g.  foo.linux2-cpython-32m.  This
>>> isn't a strong preference though.
>> 
>> I don't have a strong opionion, but placing the platform name at the
>> start is probably better to be consistent with
>> sysconfig.get_platform().
> 
> What about the architecture (i386, amd64)?  With every increase in length I
> start to get more concerned.  We could encode the platform and architecture,
> but that gets into cryptic territory.  OTOH, would you really co-install i386
> and amd64 shared libraries on the same machine?  (hello NFS ;).

I don't need this, but then again I primarily use a platform where the vendor 
has 
a proper solution for having binaries for multiple architectures ;-)

Ronald



smime.p7s
Description: S/MIME cryptographic signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New regex module for 3.2?

[Nick Coghlan]

On Fri, Jul 23, 2010 at 8:16 PM, Hrvoje Niksic  wrote:
> The performance trade-off should make regex slower with sufficiently small
> compiled regex cache, when a lot of time is wasted on compilation.  But as
> the cache gets larger (and, for fairness, of the same size in both
> implementations), regex should outperform re.  Georg, would you care to
> measure if there is a difference in performance with an even larger cache?

That's not quite accurate. The figure that matters is "matches per
compilation", and that is ultimately governed by the nature of the
application once the cache is sufficiently large (e.g. an application
that compiles 10 different regular expressions, but also only matches
each one against a single string is going to be slowed down
significantly by a switch to regex no matter how big the compilation
cache may be).

The total runtime for a particular re matching exercise is "(average
compilation time)*(number of times compiled) + (average match
time)*(number of matches performed)"

We know that the new regex module speeds up matching at the cost of
slowing down compilation. Whether that is a net increase for the
application as a whole depends on 4 things:

X: the average speed change in compilation for the application
Y: the average speed change in matching for the application
A: the proportion of time the application currently spends compiling
regular expressions
B: the proportion of time the application currently spends matching
regular expressions

The overall proportional speed change for re operations in a given
application is then just X*A + Y*B. For regex to be an improvement for
that application, the resulting figure needs to be less than 1.

For example, suppose X=2, Y=0.5 (i.e. matching is twice as fast on
average, but compilation also takes twice as long), then the overall
speed change would be 2*A + 0.5*B. For that to be a wash, the original
application needs to spend 1/3 of its expression matching time
compiling the expressions and 2/3 actually matching them (the
application with the new engine would then spend 2/3 of its time
compiling and only 1/3 matching).

That's going to be a tough assessment to make in practice, but I think
Georg's pygments test suite example shows that there is real world
code out there that already spends a lot of time on compilation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382 progress: import hooks

[Steve Holden]

On 7/23/2010 11:57 AM, Brett Cannon wrote:
> 
> 
> On Thu, Jul 22, 2010 at 19:19, P.J. Eby  > wrote:
> 
> At 01:51 PM 7/22/2010 +0100, Martin v. Löwis wrote:
> 
> At EuroPython, I sat down with Brett and we propose an approach
> how namespace packages get along with import hooks. I reshuffled
> the order in which things get done a little bit, and added a
> section that elaborates on the hooks.
> 
> Basically, a finder will need to support a find_path method,
> return all .pth files, and a loader will need to support a
> load_module_with_path method, to initialize __path__.
> 
> Please comment if you think that this needs further changes;
> 
> 
> I'm not certain I understand it precisely.  There seem to be some
> ambiguities in the spec, e.g.:
> 
> "If fullname is not found, is not a package, or does not have any
> *.pth files, None must be returned."
> 
> What does "is not a package" actually mean in that context?
> 
> 
> The module is a module but not a package.
>  
so s/is not a package/is a module rather than a package/ perhaps?

regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
DjangoCon US September 7-9, 2010http://djangocon.us/
See Python Video!   http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[schmir]

Ronald Oussoren  writes:

> On 23 Jul, 2010, at 11:54, Barry Warsaw wrote:
>
>> 
>> What about the architecture (i386, amd64)?  With every increase in length I
>> start to get more concerned.  We could encode the platform and architecture,
>> but that gets into cryptic territory.  OTOH, would you really co-install i386
>> and amd64 shared libraries on the same machine?  (hello NFS ;).
>
> I don't need this, but then again I primarily use a platform where the vendor 
> has 
> a proper solution for having binaries for multiple architectures ;-)

Well, Apple doesn't prevent people from building 32/64 bit-only python
installations. Doesn't that give you 3 choices i386, amd64, fat??
And you can have framework or non-framework builds.

Doesn't anybody else think this is lost work for very little gain? My
/usr/lib/python2.6/site-packages directory consumes 200MB on disk. I
couldn't care less if my /usr/lib/python2.5/site-packages consumed the
same amount of disk space...

- Ralf
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New regex module for 3.2?

[Georg Brandl]

Am 23.07.2010 12:20, schrieb Nick Coghlan:
> On Fri, Jul 23, 2010 at 8:16 PM, Hrvoje Niksic  wrote:
>> The performance trade-off should make regex slower with sufficiently small
>> compiled regex cache, when a lot of time is wasted on compilation.  But as
>> the cache gets larger (and, for fairness, of the same size in both
>> implementations), regex should outperform re.  Georg, would you care to
>> measure if there is a difference in performance with an even larger cache?
> 
> That's not quite accurate. The figure that matters is "matches per
> compilation", and that is ultimately governed by the nature of the
> application once the cache is sufficiently large (e.g. an application
> that compiles 10 different regular expressions, but also only matches
> each one against a single string is going to be slowed down
> significantly by a switch to regex no matter how big the compilation
> cache may be).
> 
> The total runtime for a particular re matching exercise is "(average
> compilation time)*(number of times compiled) + (average match
> time)*(number of matches performed)"
> 
> We know that the new regex module speeds up matching at the cost of
> slowing down compilation. Whether that is a net increase for the
> application as a whole depends on 4 things:
> 
> X: the average speed change in compilation for the application
> Y: the average speed change in matching for the application
> A: the proportion of time the application currently spends compiling
> regular expressions
> B: the proportion of time the application currently spends matching
> regular expressions
> 
> The overall proportional speed change for re operations in a given
> application is then just X*A + Y*B. For regex to be an improvement for
> that application, the resulting figure needs to be less than 1.
> 
> For example, suppose X=2, Y=0.5 (i.e. matching is twice as fast on
> average, but compilation also takes twice as long), then the overall
> speed change would be 2*A + 0.5*B. For that to be a wash, the original
> application needs to spend 1/3 of its expression matching time
> compiling the expressions and 2/3 actually matching them (the
> application with the new engine would then spend 2/3 of its time
> compiling and only 1/3 matching).
> 
> That's going to be a tough assessment to make in practice, but I think
> Georg's pygments test suite example shows that there is real world
> code out there that already spends a lot of time on compilation.

And for Pygments, it's not only the case for the test suite, since many
applications consist of highlighting one file/one piece of code calling
it as a program.

When using Pygments as a library, regexes are only compiled once per lexer
class, so e.g. in a web appliation that highlights files (like a repository
browser), the additional cost is negligible.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New regex module for 3.2?

[Georg Brandl]

Am 23.07.2010 11:16, schrieb Hrvoje Niksic:
> On 07/22/2010 01:34 PM, Georg Brandl wrote:
>> Timings (seconds to run the test suite):
>>
>> re 26.689  26.015  26.008
>> regex  26.066  25.797  25.865
>>
>> So, I thought there wasn't a difference in performance for this use case
>> (which is compiling a lot of regexes and matching most of them only a
>> few times in comparison).  However, I found that looking at the regex
>> caching is very important in this case: re._MAXCACHE is by default set to
>> 100, and regex._MAXCACHE to 1024.  When I set re._MAXCACHE to 1024 before
>> running the test suite, I get times around 18 (!) seconds for re.
> 
> This seems to point to re being significantly *faster* than regexp, even 
> in matching, and as such may be something the author would want to look 
> into.
> 
> Nick writes:
> 
>  > That still fits with the compile/match performance trade-off changes
>  > between re and regex though.
> 
> The performance trade-off should make regex slower with sufficiently 
> small compiled regex cache, when a lot of time is wasted on compilation. 
>   But as the cache gets larger (and, for fairness, of the same size in 
> both implementations), regex should outperform re.  Georg, would you 
> care to measure if there is a difference in performance with an even 
> larger cache?

I did measure that, and there are no significant differences in timing.

I also did the check the other way around, and restricting regex._MAXCACHE
to 100 I got from 26 seconds to 42 seconds. (Nick, is that enough data to
calculate A and B now? ;)

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2010-07-16 - 2010-07-23)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2826 open (+62) / 18349 closed (+21) / 21175 total (+83)

Open issues with patches:  1151

Average duration of open issues: 692 days.
Median duration of open issues: 480 days.

Open Issues Breakdown
   open  2780 (+60)
languishing16 ( +0)
pending29 ( +2)

Issues Created Or Reopened (89)
___

No float formatting in PyString_FromFormat 2010-07-19
   http://bugs.python.org/issue2813reopened mark.dickinson  
 
   patch   

os.curdir as the default argument for os.listdir   2010-07-23
CLOSED http://bugs.python.org/issue6095reopened tarek   
 
   patch   

weak dict iterators are fragile because of unpredictable GC ru 2010-07-23
   http://bugs.python.org/issue7105reopened mark.dickinson  
 
   patch   

subprocess.Popen documentation should contain a good warning a 2010-07-22
   http://bugs.python.org/issue7950reopened r.david.murray  
 
   patch   

code.InteractiveInterpreter fails to change locals when invoke 2010-07-16
   http://bugs.python.org/issue9274created  ericp   
 
   

python 2.7 OS X installer no longer installs /usr/local/bin sy 2010-07-16
CLOSED http://bugs.python.org/issue9275created  ned.deily   
 
   

pickle should support methods  2010-07-16
   http://bugs.python.org/issue9276created  exarkun 
 
   

test_struct failure on ARM 2010-07-16
   http://bugs.python.org/issue9277created  mark.dickinson  
 
   patch, buildbot 

rename 2to3 & pydoc to 2to3.py & pydoc.py  2010-07-16
CLOSED http://bugs.python.org/issue9278created  srid
 
   

get rid of pdb.doc 2010-07-16
CLOSED http://bugs.python.org/issue9279created  georg.brandl
 
   easy

sharedinstall target doesn't depend on sharedmods target   2010-07-17
CLOSED http://bugs.python.org/issue9280created  Arfrever
 
   patch   

Race condition in distutils.dir_util.mkpath()  2010-07-17
   http://bugs.python.org/issue9281created  Arfrever
 
   patch   

Bug in --listfuncs option of trace.py  2010-07-17
CLOSED http://bugs.python.org/issue9282created  eli.bendersky   
 
   patch   

buggy repr for os.environ  2010-07-17
   http://bugs.python.org/issue9283created  pitrou  
 
   

inspect.findsource() cannot find source for doctest code   2010-07-17
   http://bugs.python.org/issue9284created  djc 
 
   

A decorator for cProfile and profile modules   2010-07-17
   http://bugs.python.org/issue9285created  giampaolo.rodola
 
   patch, patch, needs review  

email.utils.parseaddr returns garbage for invalid input2010-07-17
   http://bugs.python.org/issue9286created  merwok  
 
   

Cosmetic fix in OtherFileTests.testOpenDir 2010-07-17
   http://bugs.python.org/issue9287created  ocean-city  
 
   patch, easy 

Disambiguate :option: and :cm

Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]

[Brett Cannon]

On Mon, Jul 12, 2010 at 02:13, Jesse Noller  wrote:

> On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum 
> wrote:
> [snip]
> > Great points Jesse! Since I really don't have the time or expertise to
> > make a judgment on this PEP, I hereby appoint you chair of the
> > approval process for this PEP. That basically means that when you
> > think it's ready to be approved, you say so, and it's a done deal. The
> > remaining feedback cycle is up to you now -- it sounds like you're
> > ready for closure, which sounds good to me (again, without having read
> > the PEP or tried to write something using the proposed code). You can
> > do it however you like: you can declare it approved now, or read it
> > over once more yourself and suggest some final changes, or set a
> > period (e.g. 48 hours) during which final comments have to be
> > received, which you then will judge by merit or by your whim, or you
> > can flip a coin or say a prayer... (I've tried most of those myself in
> > the past and haven't done too badly if I say so myself. :-) You're the
> > boss now. I know you will do the right thing for this PEP.
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> >
>
> So, after some cool down - and the last rounds of discussion which
> triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148
> "futures - execute computations asynchronously". I feel that it's a
> good addition, and a good start for something bigger down the road.
>
> Brian - you'll need to provide someone such as Martin or Georg your
> public key for ssh access into SVN, and you'll need developer access
> to the bug tracker.
>

Brian, did you ever get your keys to Martin or Georg? If not please do (or
send it to me) and let us know what your bugs.python.org username is to get
Developer privileges.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382 progress: import hooks

[P.J. Eby]

At 11:57 AM 7/23/2010 +0100, Brett Cannon wrote:


On Thu, Jul 22, 2010 at 19:19, P.J. Eby 
<p...@telecommunity.com> wrote:


What does "is not a package" actually mean in that context?


The module is a module but not a package.


Um...  that's not any clearer.  Are you saying that a module of the 
same name takes precedence over a package?  Is that the current 
precedence as well?



Regarding load_module_with_path(), how does its specification differ 
from simply creating a module in sys.modules, setting its __path__, 
and then invoking the standard load_module()? Â (i.e., is this 
method actually needed, since a correct PEP 302 loader *must* reuse 
an existing module object in sys.modules)



It must reuse the module itself but a proper reload would reset 
__path__ as leaving it unchanged is not a proper resetting of the 
module object. So this module is needed in order to force the loaderÂ


Um, no.  Reloading doesn't reset the module contents, not even 
__path__.  Never has, from Python 2.2 through 2.7 -- even in 3.1.  At 
least, not for normal filesystem .py/.pyc files.  (I tested with 
'os', adding an extra 'foo' attribute, and also setting a __path__; 
both were unaffected by reload(), in all 7 Python versions.


Perhaps you're saying this happens with zipfiles, or packages that 
already have a __path__, or...?




Â

Am I correct in understanding that, as written, one would have to 
redefine __import__ to implement this in a library for older Python 
versions? Â Or is it implementable as a meta_path importer?



Redefine __import__ (unless Martin and I are missing something, but 
I tried to think of how to implement this use sys.meta_path and 
couldn't come up with a solution).


I'm thinking it *could* be done with a meta_path hook, but only by 
doubling the search length in the event that the search failed.  That 
seems a bit icky, but replacing the entire import process seems 
ickier (more code surface to maintain, more bug potential) in the 
case of supporting older Pythons.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[Reid Kleckner]

On Fri, Jul 23, 2010 at 1:58 AM, stefan brunthaler
 wrote:
>> Do I understand correctly that you modify the byte code of modules/functions
>> at runtime?
>>
> Yes. Quickening is runtime only optimization technique that rewrites
> instructions from a generic instruction to an optimized derivative
> (orignally for the Java virtual machine). It is completely hidden from
> the compiler and has no other dependencies than the interpreter
> dispatch routine itself.

How do you generate the specialized opcode implementations?
Presumably that is done ahead of time, or you'd have to use a JIT,
which is what you're avoiding.

I'm guessing from your comments below about cross-module inlining that
you generate a separate .c file with the specialized opcode bodies and
then call through to them via a table of function pointers indexed by
opcode, but I could be totally wrong.  :)

> Another benefit of using my technique is that a compiler could decide
> to inline all of the functions of the optimized derivatives (e.g., the
> float_add function call inside my FLOAT_ADD interpreter instruction).
> Unfortunately, however, gcc currently does not allow for cross-module
> inlining (AFAIR). (Preliminary tests with manually changing the
> default inlining size for ceval.c resulted in speedups of up to 1.3 on
> my machine, so I think inlinling of function bodies for the optimized
> derivatives would boost performance noticeably.)

There are a variety of solutions to getting cross-module inlining
these days.  Clang+LLVM support link-time optimization (LTO) via a
plugin for gold.  GCC has LTO and LIPO as well.

>> Such an approach would also be very useful for Cython. Think of a profiler
>> that runs a program in CPython and tells you exactly what static type
>> annotations to put where in your Python code to make it compile to a fast
>> binary with Cython. Or, even better, it could just spit out a .pxd file that
>> you drop next to your .py file and that provides the static type information
>> for you.

This would be interesting.  We have (obviously) have similar
instrumentation in unladen swallow to gather type feedback.  We talked
with Craig Citro about finding a way to feed that back to Cython for
exactly this reason, but we haven't really pursued it.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include datetime.py in stdlib or not?

[Alexander Belopolsky]

Thanks, everyone who followed up here and on the tracker.  I am
readying the patch for check in, but as I look back through the
messages, I don't really see anyone's answer to the question in the
subject:

*  Include datetime.py in stdlib or not?

I hope this means an implied "yes, include."  Since committing the
patch will case a bit of file movement, I would like to pre-announce
the commit in case my timing is not right.

Here is the link to the issue: http://bugs.python.org/issue7989 .
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include datetime.py in stdlib or not?

[Benjamin Peterson]

2010/7/23 Alexander Belopolsky :
> Thanks, everyone who followed up here and on the tracker.  I am
> readying the patch for check in, but as I look back through the
> messages, I don't really see anyone's answer to the question in the
> subject:
>
> *  Include datetime.py in stdlib or not?
>
> I hope this means an implied "yes, include."  Since committing the
> patch will case a bit of file movement, I would like to pre-announce
> the commit in case my timing is not right.

It's a bit hard to get a strong affirmative because it's not really
anyone's decision. I would say, though, given that there haven't been
any strong -1s, you can go ahead.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[Cesare Di Mauro]

2010/7/23 stefan brunthaler 

> > This sounds like wpython (a CPython derivative with a wider set of byte
> code
> > commands) could benefit from it.
> >
> I am aware of the wpython project of Cesare di Mauro.


wpython has reached 1.1 final version. If you are interested, you can find
it here: http://code.google.com/p/wpython2/ and you can download the new
slides that cover the improvements over 1.0 alpha.


> I change the
> instruction format from bytecode to wordcode, too (because it allows
>  for more efficient instruction decoding).


Did you used wpython wordcode format, or a new one?


> Contrary to his approach,
> however, I do not change the instruction encoding to pack in
> additional optimizations. (I hope to have put that correctly; I have
>  seen his slides about a year ago.)
>

Yes, you're right. wpython approach is to encode as much information as it
can to save space, decoding time, "specialize" some opcodes, etc..

Cesare
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[stefan brunthaler]

> How do you generate the specialized opcode implementations?
>
I have a small code generator written in Python that uses Mako
templates to generate C files that can be included in the main
interpreter. It is a data driven approach that uses type information
gathered by gdb and check whether given types implement for instance a
nb_add method.

> Presumably that is done ahead of time, or you'd have to use a JIT,
> which is what you're avoiding.
>
Yes, and yes: I execute the code generator before compiling the Python
interpreter, and I am interested in purely interpretative optimization
techniques.


> I'm guessing from your comments below about cross-module inlining that
> you generate a separate .c file with the specialized opcode bodies and
> then call through to them via a table of function pointers indexed by
> opcode, but I could be totally wrong.  :)
>
No, dead on ;)
Probably a small example from the top of my head illustrates what is going on:

TARGET(FLOAT_ADD):
  w= POP();
  v= TOP();
  x= PyFloat_Type.tp_as_number->nb_add(v, w);
  SET_TOP(x);
  if (x != NULL) FAST_DISPATCH();
  break;

And I extend the standard indirect threaded code dispatch table to
support the FLOAT_ADD operation.


> There are a variety of solutions to getting cross-module inlining
> these days.  Clang+LLVM support link-time optimization (LTO) via a
> plugin for gold.  GCC has LTO and LIPO as well.
>
A PhD colleague from our institute pointed the gold stuff out to me
yesterday, I am going to check out if any of these solutions would
work. A deeper problem here is that the heuristics of the compilers
are ill-suited to the needs of compiling an interpreter dispatch
routine -- I will investigate this further in future research.



> This would be interesting.  We have (obviously) have similar
> instrumentation in unladen swallow to gather type feedback.  We talked
> with Craig Citro about finding a way to feed that back to Cython for
> exactly this reason, but we haven't really pursued it.
>
Ok; I think it would actually be fairly easy to use the type
information gathered at runtime by the quickening approach. Several
auxiliary functions for dealing with these types could be generated by
my code generator as well. It is probably worth looking into this,
though my current top-priority is my PhD research, so I cannot promise
to being able to allocate vast amounts of time for such endeavours.


Best,
--stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[stefan brunthaler]

> wpython has reached 1.1 final version. If you are interested, you can find
> it here: http://code.google.com/p/wpython2/ and you can download the new
> slides that cover the improvements over 1.0 alpha.
>
Thanks for the hint, I will definitely check your new slides.


> Did you used wpython wordcode format, or a new one?
>
No, actually I was well into working on my stuff when you announced
wpython last year. My latest instruction format uses a native machine
word (64bit) that contains two 32bit halves with the opcode in the
lower half and the operand in the upper half. While the opcode
certainly does not exceed 10bits or so, I need more than a byte to
support more operations (my latest interpreter has 395 instructions).
Our instruction decoding is almost identical, though.


> Yes, you're right. wpython approach is to encode as much information as it
> can to save space, decoding time, "specialize" some opcodes, etc..
>
Yes, I see that wpython eliminates common instruction sequences. From
my understanding, it corresponds to using static superinstructions in
combination with a changed instruction format. Aside of the
optimizations in the operation implementation wpython allows to
eliminate some instruction dispatches, which are notoriously
inefficient. I think it is a very nice approach and some form of
inline caching with quickening might well boost performance even more.

Cheers,
--stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[Reid Kleckner]

On Fri, Jul 23, 2010 at 11:26 AM, stefan brunthaler
 wrote:
>> I'm guessing from your comments below about cross-module inlining that
>> you generate a separate .c file with the specialized opcode bodies and
>> then call through to them via a table of function pointers indexed by
>> opcode, but I could be totally wrong.  :)
>>
> No, dead on ;)
> Probably a small example from the top of my head illustrates what is going on:
>
> TARGET(FLOAT_ADD):
>  w= POP();
>  v= TOP();
>  x= PyFloat_Type.tp_as_number->nb_add(v, w);
>  SET_TOP(x);
>  if (x != NULL) FAST_DISPATCH();
>  break;
>
> And I extend the standard indirect threaded code dispatch table to
> support the FLOAT_ADD operation.

I think I was wrong, but now I understand.  The inlining you want is
to get the nb_add body, not the opcode body.

The example you've given brings up a correctness issue.  It seems
you'd want to add checks that verify that the operands w and v are
both floats, and jump to BINARY_ADD if the guard fails.  It would
require reshuffling the stack operations to defer the pop until after
the check, but it shouldn't be a problem.

>> This would be interesting.  We have (obviously) have similar
>> instrumentation in unladen swallow to gather type feedback.  We talked
>> with Craig Citro about finding a way to feed that back to Cython for
>> exactly this reason, but we haven't really pursued it.
>>
> Ok; I think it would actually be fairly easy to use the type
> information gathered at runtime by the quickening approach. Several
> auxiliary functions for dealing with these types could be generated by
> my code generator as well. It is probably worth looking into this,
> though my current top-priority is my PhD research, so I cannot promise
> to being able to allocate vast amounts of time for such endeavours.

I think you also record (via gdb) exactly the information that we
record.  I now see three consumers of type feedback from the CPython
interpreter: you or any quickening approach, Cython, and Unladen
Swallow.  It might be worth converging on a standard way to record
this information and serialize it so that it can be analyzed offline.

Our feedback recording mechanism currently uses LLVM data structures,
but the point of quickening is to avoid that kind of dependency, so
we'd need to rewrite it before it would really be useful to you.

Reid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 optimizations...

[stefan brunthaler]

> I think I was wrong, but now I understand.  The inlining you want is
> to get the nb_add body, not the opcode body.
>
Exactly. This would increase performace by quite a bit -- I will start
experimentation with that stuff a.s.a.p.


> The example you've given brings up a correctness issue.  It seems
> you'd want to add checks that verify that the operands w and v are
> both floats, and jump to BINARY_ADD if the guard fails.  It would
> require reshuffling the stack operations to defer the pop until after
> the check, but it shouldn't be a problem.
>
That's usually the problem when you're doing something from the top of
your head, especially when its already 9pm :)
You're right, a simple way around this is either:

TARGET(FLOAT_ADD):
   if (!(TOP()->ob_type == SECOND()->ob_type && TOP()->ob_type ==
&PyFloat_Type))
  goto TARGET_BINARY_ADD_SKIP_DECODE;
   ...remains the same...

Note: the skip_decode is necessary because otherwise it would advance
the instruction pointer.
Another, simple approach is to add another set of labels to denote
inline cache misses, e.g.:

TARGET(BINARY_ADD):
   w= POP();
   v= TOP();
  BINARY_ADD_CACHE_MISS:
   x= PyNumber_Add(v, w);
   ...

TARGET(FLOAT_ADD):
   w= POP();
   v= TOP();
   if (v->ob_type != w->ob_type || v->ob_type != &PyFloat_Type)
 goto BINARY_ADD_CACHE_MISS;
   ...

You could also call the PyNumber_Add function yourself instead of the
goto, but I did not implement it this way...



> I think you also record (via gdb) exactly the information that we
> record.  I now see three consumers of type feedback from the CPython
> interpreter: you or any quickening approach, Cython, and Unladen
> Swallow.  It might be worth converging on a standard way to record
> this information and serialize it so that it can be analyzed offline.
>
Indeed. Probably a bigger set of types of frequently used C extension
modules would be useful as well. I am doing the simplest possible
thing here: since the gdb pretty print already is pretty close to a
Python data type definition, I just copied this and did a few search
and replace commands to convert it to a valid Python data type.


> Our feedback recording mechanism currently uses LLVM data structures,
> but the point of quickening is to avoid that kind of dependency, so
> we'd need to rewrite it before it would really be useful to you.
>
Ok.


--stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jul 23, 2010, at 12:54 PM, Barry Warsaw wrote:

>On Jul 23, 2010, at 11:48 AM, Ronald Oussoren wrote:
>
>>>  I'd be open to adding the
>>> platform name to the tag, but I'd probably define it as part of the
>>> implementation field, e.g. foo.cpython-linux2-32m.so.  Or maybe
>>> start with the platform name, e.g.  foo.linux2-cpython-32m.  This
>>> isn't a strong preference though.
>>
>>I don't have a strong opionion, but placing the platform name at the
>>start is probably better to be consistent with
>>sysconfig.get_platform().
>
>What about the architecture (i386, amd64)?  With every increase in
>length I start to get more concerned.  We could encode the platform
>and architecture, but that gets into cryptic territory.  OTOH, would
>you really co-install i386 and amd64 shared libraries on the same
>machine?  (hello NFS ;).

Thinking about this some more, I'd rather *not* include the platform or
architecture in the tag by default.  They aren't really necessary to support
the instigating use case and will probably be fairly uncommon.

I'd be okay including a configure option to allow you to add whatever you want
after the implementation, version, and flags.  E.g. something like:

./configure --with-abi-tag-extension=linux2

would lead to foo.cpython-32m-linux2.so, so not the nicer names we'd prefer
but probably good enough for your purposes.

Would that work for you?
-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jul 23, 2010, at 01:46 PM, sch...@gmail.com wrote:

>Doesn't anybody else think this is lost work for very little gain? My
>/usr/lib/python2.6/site-packages directory consumes 200MB on disk. I
>couldn't care less if my /usr/lib/python2.5/site-packages consumed the
>same amount of disk space...

Right, you probably don't care now that your extension modules live in foo.so
so it probably won't make much difference if they were named foo-blahblah.so,
as long as they import. :)

If you use Debian or Ubuntu though, it'll be a win for you by allow us to make
Python support much more robust.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Barry Warsaw]

On Jul 23, 2010, at 08:56 PM, Nick Coghlan wrote:

>On Fri, Jul 23, 2010 at 12:40 AM, Barry Warsaw 
>wrote:
>> Python implementations *MAY* include additional flags in the file
>> name tag as appropriate.  For example, on POSIX systems these flags
>> will also contribute to the file name:
>>
>>  * ``--with-pydebug`` (flag: ``d``)
>>  * ``--with-pymalloc`` (flag: ``m``)
>>  * ``--with-wide-unicode`` (flag: ``u``)
>>
>> By default in Python 3.2, ``configure`` enables ``--with-pymalloc``
>> so shared library file names would appear as ``foo.cpython-32m.so``.
>> When the other two flags are also enabled, the file names would be
>> ``foo.cpython-32dmu.so``.
>>
>> (This PEP only addresses build issues on POSIX systems that use the
>> ``configure`` script.  While Windows or other platform support is not
>> explicitly disallowed under this PEP, platform expertise is needed in
>> order to evaluate, describe, and implement support on such
>> platforms.)
>
>This leads me to a question: how do these configure options affect the
>PEP 384 stable ABI? That PEP is currently silent on the issue, while
>PEP 3149 appears to implicitly assume that "abi3" completely specifies
>the ABI.

It's a great question - perhaps Martin can chime in?  It may be that 'abiX'
isn't enough to fully specify compatible extension modules even when that
module is written entirely and solely against PEP 384.  In that case, we may
need to include the configure flags in the tag, e.g. foo.abi3-dmu.so.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] http://bugs.python.org/issue231540

[Mark Lawrence]

Is there any money to pay for the forthcoming 10th birthday party for 
this issue?  Is the OP still alive?


Kindest regards.

Mark Lawrence.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Paul Moore]

On 23 July 2010 23:26, Mark Lawrence  wrote:
> Is there any money to pay for the forthcoming 10th birthday party for this
> issue?  Is the OP still alive?

I'm not sure the sarcasm helps much. What do you suggest should be
done with the request? Nobody has provided a patch, so there's nothing
to commit. Closing it as "won't fix" seems unreasonable, as I imagine
that should a suitable patch be supplied, it would be accepted.

There's no magical means by which such a patch would appear, though.
The OP clearly[1] is either not interested enough or doesn't have the
skills to provide a patch, and no-one else has stepped up to do so.

Note that it's been classified as a feature request, not a bug. So
there's nothing wrong, as such, with it remaining unresolved.

Paul.

[1] I say "clearly" - it may be that he could provide a patch if
asked. Maybe it would be worth you contacting him to ask if the issue
is still a problem for him, and whether he can assist in resolving it.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python profiler and other tools

[Alexander Belopolsky]

I am changing the subject from "http://bugs.python.org/issue231540,";
because if there was a prize for a non-descriptive subject, OP would
win it.

There must be a good reason why traditional software development tools
such as debugger, profiler and coverage are mostly neglected in
python.  Terry and I have recently discovered that the trace
(coverage) module does not have any unit tests [1] and pretty much
does not work in 3.x. [2]  One of the recently fixed trace module
problems [3] is likely affecting profile module as well.   I tried to
figure out who works on profile to add them to the nosy list and could
not find anyone.

I seem to remember someone suggesting that python's dynamic nature
makes traditional development tools unnecessary.   A print statement
is the only debugging tool that you need.  There may be some truth to
it, but in this case, python should not distribute these tools.  I am
personally in the opposite camp: I love pdb and I am a big believer in
profiling and coverage testing.   I have entered myself as a
maintainer for the trace module which is the simplest of the three,
but I would like to learn from developers of pdb and profile on a
range of overlapping issues.

[1] http://bugs.python.org/issue9315
[2] http://bugs.python.org/issue9317
[3] http://bugs.python.org/issue9323

On Fri, Jul 23, 2010 at 6:26 PM, Mark Lawrence  wrote:
> Is there any money to pay for the forthcoming 10th birthday party for this
> issue?  Is the OP still alive?
>
> Kindest regards.
>
> Mark Lawrence.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/alexander.belopolsky%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Mark Lawrence]

On 24/07/2010 00:09, Paul Moore wrote:

On 23 July 2010 23:26, Mark Lawrence  wrote:

Is there any money to pay for the forthcoming 10th birthday party for this
issue?  Is the OP still alive?


I'm not sure the sarcasm helps much. What do you suggest should be
done with the request? Nobody has provided a patch, so there's nothing
to commit. Closing it as "won't fix" seems unreasonable, as I imagine
that should a suitable patch be supplied, it would be accepted.

There's no magical means by which such a patch would appear, though.
The OP clearly[1] is either not interested enough or doesn't have the
skills to provide a patch, and no-one else has stepped up to do so.

Note that it's been classified as a feature request, not a bug. So
there's nothing wrong, as such, with it remaining unresolved.

Paul.

[1] I say "clearly" - it may be that he could provide a patch if
asked. Maybe it would be worth you contacting him to ask if the issue
is still a problem for him, and whether he can assist in resolving it.


Paul,

I'm on the verge of giving up my time because the whole system is a 
complete and utter waste of my time.  I feel quite happy that in my 
brief tenure I've closed 46 issues, but there's so many more that could 
have been closed, but yet again you don't even get the courtesy of a 
response when there's more in the pipeline that could be closed.  I'd 
quote the issue numbers here and now, but I'm just too flaming tired to 
do so, though a quick count indicates I've got 23 ongoing that I'm 
attempting to sort.


As it happens, I have been having discussions offline in an attempt to 
shift the culture of Python development but I don't believe that 
anything will come out of it.  Let's face it, development is much more 
interesting than bug fixes. And once again, if some stupid idiot 
volunteer bothers to put in a patch to the code and/or the unit test, 
and it sits and rots for five years, is that person likely to come back 
to Python?  Strangely, some do.


Sorry, I'm off to bed.

Yours feeling most disillusioned with python-dev.

Mark Lawrence.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Terry Reedy]

On 7/23/2010 6:26 PM, Mark Lawrence wrote:

Is there any money to pay for the forthcoming 10th birthday party for
this issue? Is the OP still alive?


This reminds me of some low priority items on my todo list ;-)

Hmmm. Right now there are 756 open feature requests out of 2779 open 
issues. There have been bug days. Maybe after 3.2 is out, we should have 
a feature day to see if some feature requests cannot be closed as 
obsolete, unwise, unwanted, or too difficult.


It is possible that PEP3099, Rejected ideas, could be expanded to help 
triagers close already rejected ideas.


Something I/someone might do is make some manageable lists of related 
ideas (say, additions to builtin functions) and post them on 
python-ideas for discussion and voting.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [isssue 2001] Pydoc enhancement patch questions

[Ron Adam]

This regards the following feature request.

http://bugs.python.org/issue2001

Summery:

The pydoc "gui" enhancement patch adds a navigation bar on each page with 
features that correspond to abilities that are currently available to help. 
 ie.. a get field, to get help on an item, a find field, to search for 
modules by keyword, plus modules, topics, and keyword index links.


The file links read the python file in as text and inserts it into a web 
page instead of relying on the browser to do the right thing. (often enough 
it doesn't)


To achieve this I reworked the existing pydoc server into one that can 
respond to a navigation bar at the top of the served pages.  The new 
"local_text_server" will exist in the http package where the other server 
related modules are.  The callback function passed to the server does all 
the pydoc specific work so the server itself is a simple general purpose 
text server useful for making browser interface front ends.


This also removed the need for tkinter in pydoc and allows pydoc -g to work 
on computers where tk isn't installed.



As per discussion on the tracker:


Alexander Belopolsky  added the comment:

+:program:`pydoc` :option:`-g` will start the server and additionally open a web
+browser to a module index page.  Each served page has a navagation bar at the
+top where you can 'get' help on a individual item, 'find' all modules with a
+keyword in thier synopsis line, and goto indexes for 'modules', 'topics' and
+'keywords'.

I am not sure I like the fact that the browser is started automatically.
Please bring this up on python-dev.  This may be an opportunity to
rethink pydoc command line switches.  For example, -p and -g are
currently exclusive, but it would make sense for -g to start server on
the port specified by -p.


So are any thoughts on starting the web browser automatically, and on how 
the -g and -p command line switches work?


I'm also not sure about the name for the server.  I've use 
local_text_server, but it may also be useful in non-local cases as well.


The newest patch is...

 http://bugs.python.org/file18165/pydoc_server3.diff

Any feedback will be welcome.

Ron








___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Simon de Vlieger]

On 24 July 2010 01:39, Mark Lawrence  wrote:
> On 24/07/2010 00:09, Paul Moore wrote:
>>
>> On 23 July 2010 23:26, Mark Lawrence  wrote:
>>>
>>> Is there any money to pay for the forthcoming 10th birthday party for
>>> this
>>> issue?  Is the OP still alive?
>>
>> I'm not sure the sarcasm helps much. What do you suggest should be
>> done with the request? Nobody has provided a patch, so there's nothing
>> to commit. Closing it as "won't fix" seems unreasonable, as I imagine
>> that should a suitable patch be supplied, it would be accepted.
>>
>> There's no magical means by which such a patch would appear, though.
>> The OP clearly[1] is either not interested enough or doesn't have the
>> skills to provide a patch, and no-one else has stepped up to do so.
>>
>> Note that it's been classified as a feature request, not a bug. So
>> there's nothing wrong, as such, with it remaining unresolved.
>>
>> Paul.
>>
>> [1] I say "clearly" - it may be that he could provide a patch if
>> asked. Maybe it would be worth you contacting him to ask if the issue
>> is still a problem for him, and whether he can assist in resolving it.
>
> Paul,
>
> I'm on the verge of giving up my time because the whole system is a complete
> and utter waste of my time.  I feel quite happy that in my brief tenure I've
> closed 46 issues, but there's so many more that could have been closed, but
> yet again you don't even get the courtesy of a response when there's more in
> the pipeline that could be closed.  I'd quote the issue numbers here and
> now, but I'm just too flaming tired to do so, though a quick count indicates
> I've got 23 ongoing that I'm attempting to sort.
>
> As it happens, I have been having discussions offline in an attempt to shift
> the culture of Python development but I don't believe that anything will
> come out of it.  Let's face it, development is much more interesting than
> bug fixes. And once again, if some stupid idiot volunteer bothers to put in
> a patch to the code and/or the unit test, and it sits and rots for five
> years, is that person likely to come back to Python?  Strangely, some do.
>
> Sorry, I'm off to bed.
>
> Yours feeling most disillusioned with python-dev.
>
> Mark Lawrence.

Mark,

when I read your emails it seems to me as if you have the greatest
concern with improving Python, the language and improving the state of
the bug tracker.

This is a great thing, people like you are much needed. However, I do
seem to notice you try to take a business-like approach here on this
mailinglist. Most people on python-dev are volunteers who (like you)
spend their free time helping and working on Python.

People who work in their free time are less likely to feel obliged to
respond immediately to an issue. They are also less likely to keep
paying attention to the bugs they were assigned.

I think a person like you is needed, someone who weeds through the
rotting bug reports (not the feature requests) and tries to follow up
on them. Is the issue persistent for the user, has it been fixed as
collateral on another fix, etcetera.

However, I think you would get more done if you switched from a
business philosophy to accepting that most people here are volunteers,
don't try to pressure volunteers. Try to do the best *you* can within
the community and let that help the project further.

Oh, and with business philosophy I mean: mails like the one you start
this thread with are interpreted by me as being very pushy, overly
sarcastic and if my project manager at the office sends me an email
like that I know I have to do it right now. I would dislike to be
spoken to like this in an voluntary environment. Do note that I do
understand where your feelings come from.

Regards,

Simon de Vlieger

P.S. a feature day sounds like a great idea!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] http://bugs.python.org/issue231540

[geremy condra]

On Fri, Jul 23, 2010 at 5:18 PM, Simon de Vlieger  wrote:
> On 24 July 2010 01:39, Mark Lawrence  wrote:
>> On 24/07/2010 00:09, Paul Moore wrote:
>>>
>>> On 23 July 2010 23:26, Mark Lawrence  wrote:

 Is there any money to pay for the forthcoming 10th birthday party for
 this
 issue?  Is the OP still alive?
>>>
>>> I'm not sure the sarcasm helps much. What do you suggest should be
>>> done with the request? Nobody has provided a patch, so there's nothing
>>> to commit. Closing it as "won't fix" seems unreasonable, as I imagine
>>> that should a suitable patch be supplied, it would be accepted.
>>>
>>> There's no magical means by which such a patch would appear, though.
>>> The OP clearly[1] is either not interested enough or doesn't have the
>>> skills to provide a patch, and no-one else has stepped up to do so.
>>>
>>> Note that it's been classified as a feature request, not a bug. So
>>> there's nothing wrong, as such, with it remaining unresolved.
>>>
>>> Paul.
>>>
>>> [1] I say "clearly" - it may be that he could provide a patch if
>>> asked. Maybe it would be worth you contacting him to ask if the issue
>>> is still a problem for him, and whether he can assist in resolving it.
>>
>> Paul,
>>
>> I'm on the verge of giving up my time because the whole system is a complete
>> and utter waste of my time.  I feel quite happy that in my brief tenure I've
>> closed 46 issues, but there's so many more that could have been closed, but
>> yet again you don't even get the courtesy of a response when there's more in
>> the pipeline that could be closed.  I'd quote the issue numbers here and
>> now, but I'm just too flaming tired to do so, though a quick count indicates
>> I've got 23 ongoing that I'm attempting to sort.
>>
>> As it happens, I have been having discussions offline in an attempt to shift
>> the culture of Python development but I don't believe that anything will
>> come out of it.  Let's face it, development is much more interesting than
>> bug fixes. And once again, if some stupid idiot volunteer bothers to put in
>> a patch to the code and/or the unit test, and it sits and rots for five
>> years, is that person likely to come back to Python?  Strangely, some do.
>>
>> Sorry, I'm off to bed.
>>
>> Yours feeling most disillusioned with python-dev.
>>
>> Mark Lawrence.
>
> Mark,
>
> when I read your emails it seems to me as if you have the greatest
> concern with improving Python, the language and improving the state of
> the bug tracker.
>
> This is a great thing, people like you are much needed. However, I do
> seem to notice you try to take a business-like approach here on this
> mailinglist. Most people on python-dev are volunteers who (like you)
> spend their free time helping and working on Python.
>
> People who work in their free time are less likely to feel obliged to
> respond immediately to an issue. They are also less likely to keep
> paying attention to the bugs they were assigned.
>
> I think a person like you is needed, someone who weeds through the
> rotting bug reports (not the feature requests) and tries to follow up
> on them. Is the issue persistent for the user, has it been fixed as
> collateral on another fix, etcetera.
>
> However, I think you would get more done if you switched from a
> business philosophy to accepting that most people here are volunteers,
> don't try to pressure volunteers. Try to do the best *you* can within
> the community and let that help the project further.
>
> Oh, and with business philosophy I mean: mails like the one you start
> this thread with are interpreted by me as being very pushy, overly
> sarcastic and if my project manager at the office sends me an email
> like that I know I have to do it right now. I would dislike to be
> spoken to like this in an voluntary environment. Do note that I do
> understand where your feelings come from.
>
> Regards,
>
> Simon de Vlieger
>
> P.S. a feature day sounds like a great idea!

I don't care if he yells and rants and raves, he's a volunteer too, he
does a ton of work that nobody else has stepped up to do, and IMO
we're very lucky to have him doing it.

Geremy Condra
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Set the namespace free!

[Gregory P. Smith]

On Thu, Jul 22, 2010 at 9:24 AM,  wrote:

>  I agree with the idea, but a far less radical change is needed to get the
> desired result.
> The basic idea is this: it should be possible to use any name as an
> identifier in the syntax, including names
> like 'while' and 'import'. But there is no need to mess up the entire
> language to allow this
> (either by quoting all the identifiers, perl-style, or by marking the
> keywords).
>

Yuck.  Anyone who feels they need a variable named the same a reserved word
simply feels wrong and needs reeducation.  New words are introduced very
rarely and we do care about the ramifications when we do it.

What next?  An optional way to support case insensitive names using a
unicode character prefix?

-gps


>
> All that is needed is something like this:
>
> foo = 7
> :foo = 7   # exactly like foo=7
> :while= 3  # assigns 3 to variable 'while'
> globals()['while']=3  # current way to do this
>
> print element.:for  # from example below
> #
> # keyword parameters to a function call:
> #
> BuildUrlQuery( lang='en', item='monsoon', :class='normal')  # ->
> "?lang=en&query=monsoon&class=normal"
>
>
> The generic keyword function call is a really nice language feature, but
> it's rather impaired by the need to avoid
> those names which happen to be keywords.
>
> The lack of this is most painful when you are auto-generating python code
> which forms a bridge to another language with its own
> namespace (as in XML example). It's a pain when some of the names you might
> generate could conflict with python keywords.
> So, you end up using dicts and getattrs for everything and the code gets
> much less readable. With a simple escape like :name available,
> it's worthwhile to do everything with identifiers and generate the escape
> only as needed for these.
>
>
> One of the great strengths of python is the ability to form cleans and
> comprehensive bridges to other languages and environments (thus,
> in many cases, avoiding the need to write programs in those other
> environments :-) ). This feature would fill a gap there.
>
> The python tcl/tk interface is a bit messed up since tcl/tk uses some names
> for options which conflict with python keywords,
> and so you need to add '_' to those names.
>
> There is a feature like this in VHDL:  \name\ and \while\ are identifiers,
> the backslashes are not part of the name, but just
> quote it. In  VHDL you can write identifiers like  \22\, and
> \!This?is=Strange\ as well; since VHDL creates modules that
> have named ports, and those modules can interface to things generated by
> other environments, they needed a
> way to assign any name to a port.
>
> For python, I'm not sure it makes sense to allow identifiers that doesn't
> follow the basic rule "[A-Za-z_][A-Za-z_0-9]*" -- that could
> break some debugging tools which expect variable names to be well-formed --
> but it would be useful
> to be able to use any well-formed name as an identifier, including those
> which happen to be keywords.
>
> I've suggested :name, which doesn't break old code, and doesn't require
> using any new punctuation. Syntax would not change,
> just the lexical definition of 'identifier'.  If the intent is to allow
> arbitrary names (not just well-formed ones), then n'name' would
> work better (and is consistent with existing stuff).
>
>
>
> > Date: Thu, 22 Jul 2010 10:41:39 -0400
> > From: jnol...@gmail.com
> > To: bartosz-tarnow...@zlotniki.pl
> > CC: python-dev@python.org
> > Subject: Re: [Python-Dev] Set the namespace free!
>
> >
> > On Thu, Jul 22, 2010 at 10:04 AM, Bartosz Tarnowski
> >  wrote:
> > >
> > > Hello, guys.
> > >
> > > Python has more and more reserved words over time. It becomes quite
> > > annoying, since you can not use variables and attributes of such names.
> > > Suppose I want to make an XML parser that reads a document and returns
> an
> > > object with attributes corresponding to XML element attributes:
> > >
> > >> elem = parse_xml("")
> > >> print elem.param
> > > boo
> > >
> > > What should I do then, when the attribute is a reserver word? I could
> use
> > > trailing underscore, but this is quite ugly and introduces ambiguity.
> > >
> > >> elem = parse_xml("")
> > >> print elem.for_ #?
> > >> elem = parse_xml("")
> > >> print elem.for__ #?
> > >
> > > My proposal: let's make a syntax change.
> > >
> > > Let all reserved words be preceded with some symbol, i.e. "!"
> (exclamation
> > > mark). This goes also for standard library global identifiers.
> > >
> > > !for boo in foo:
> > >!if boo is !None:
> > >!print(hoo)
> > >!else:
> > >!return !sorted(woo)
> > >
> > >
> > > This would allow the user to declare any identifier with any name:
> > >
> > > for = with(return) + try
> > >
> > > What do you think of it? It is a major change, but I think Python needs
> it.
> > >
> > > --
> > > haael
> > >
> >
> > I'm not a fan of this - I'd much prefer[1] that we use the exclamation
> > point to determine scope:
> >

Re: [Python-Dev] http://bugs.python.org/issue231540

[Brian Curtin]

On Fri, Jul 23, 2010 at 18:39, Mark Lawrence wrote:

> On 24/07/2010 00:09, Paul Moore wrote:
>
>> On 23 July 2010 23:26, Mark Lawrence  wrote:
>>
>>> Is there any money to pay for the forthcoming 10th birthday party for
>>> this
>>> issue?  Is the OP still alive?
>>>
>>
>> I'm not sure the sarcasm helps much. What do you suggest should be
>> done with the request? Nobody has provided a patch, so there's nothing
>> to commit. Closing it as "won't fix" seems unreasonable, as I imagine
>> that should a suitable patch be supplied, it would be accepted.
>>
>> There's no magical means by which such a patch would appear, though.
>> The OP clearly[1] is either not interested enough or doesn't have the
>> skills to provide a patch, and no-one else has stepped up to do so.
>>
>> Note that it's been classified as a feature request, not a bug. So
>> there's nothing wrong, as such, with it remaining unresolved.
>>
>> Paul.
>>
>> [1] I say "clearly" - it may be that he could provide a patch if
>> asked. Maybe it would be worth you contacting him to ask if the issue
>> is still a problem for him, and whether he can assist in resolving it.
>>
>
> Paul,
>
> I'm on the verge of giving up my time because the whole system is a
> complete and utter waste of my time.  I feel quite happy that in my brief
> tenure I've closed 46 issues, but there's so many more that could have been
> closed, but yet again you don't even get the courtesy of a response when
> there's more in the pipeline that could be closed.  I'd quote the issue
> numbers here and now, but I'm just too flaming tired to do so, though a
> quick count indicates I've got 23 ongoing that I'm attempting to sort.
>
> As it happens, I have been having discussions offline in an attempt to
> shift the culture of Python development but I don't believe that anything
> will come out of it.  Let's face it, development is much more interesting
> than bug fixes. And once again, if some stupid idiot volunteer bothers to
> put in a patch to the code and/or the unit test, and it sits and rots for
> five years, is that person likely to come back to Python?  Strangely, some
> do.
>
> Sorry, I'm off to bed.
>
> Yours feeling most disillusioned with python-dev.


Mark,

First off, thanks for your time. One thing I don't feel is very beneficial
is to focus so much on the number of issues you are able to close, because
that number by itself is too high-level. Closing 10 issues might not be
better than closing 5 issues. If you follow baseball, it's like a pitcher's
win total. A higher number doesn't really mean the pitcher is better, and a
lower number doesn't mean the pitcher is worse. You find the better pitcher
by drilling down into more specific statistics.

Closing an obviously invalid issue takes little effort, but it's a -1 for
the issue total. Closing an issue as fixed might take a little more effort
in tracking down a revision number, but it's a -1 for the total and some
varying value for performance, coverage, etc. Closing an issue early because
no one commented or no one proposed a patch in some certain time span is, to
me, +1 to the total, possibly more. Not closing an issue is a +/- 0.

An issue is an issue once confirmed, regardless of a lack of comment or a
lack of attention from core development. I agree that it's unfortunate
having issues drag on, but time is one of the biggest bottlenecks in the
development of open source software. We have plenty of qualified, quality,
smart people working on Python, but there are only so many hours in the day
that we can spend on it.

With that said, I hope you'll continue your efforts. What helped me when I
started out in your position was to pick issues I knew I could work on with
relative ease and come out without having to duck too many times. I knew I
wasn't going to remove the GIL, but I could start with some of the low
hanging fruit like making zipfile.ZipFile work as a context manager (first
thing I fixed, I think). Like you, I wanted to make an impact, and in order
to make a longer term impact I wanted to have some early success to get
further into the game. No one wants to keep going if it's not fun, and this
should be fun.

My suggestion is to throw more effort into less issues. Going back to
baseball, if you can advance the runners without getting on base yourself
(e.g., a sacrifice fly out), that's still a good thing for the team.
Providing a code review to a review-less issue is more valuable than
requesting that someone else provide comments -- a review moves the issue
along. Updating a patch to a current revision is worth more than seeing if
anyone still runs AIX -- updating moves the issue along.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Steven D'Aprano]

On Sat, 24 Jul 2010 12:42:12 pm Brian Curtin wrote:

> First off, thanks for your time. One thing I don't feel is very
> beneficial is to focus so much on the number of issues you are able
> to close, because that number by itself is too high-level. Closing 10
> issues might not be better than closing 5 issues. If you follow
> baseball, it's like a pitcher's win total.
[...]
> My suggestion is to throw more effort into less issues. Going back to
> baseball

Mark is from the UK. He has probably heard of baseball, but otherwise 
baseball analogies will probably go right over his head (as they went 
over mine).


-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] http://bugs.python.org/issue231540

[Steven D'Aprano]

On Sat, 24 Jul 2010 09:39:49 am Mark Lawrence wrote:

> I'm on the verge of giving up my time because the whole system is a
> complete and utter waste of my time.
[...]

Mark, you have entirely avoided the issues Paul raised. It's one thing 
to fire off a snarky email implying that the Python developers are lazy 
slackers for allowing an issue to approach ten years old, but Paul's 
response is valid. What do you suggest should be done with it? Close it 
as "Won't fix"? That doesn't help anyone. If you have the appropriate 
patch and unit tests, then great, you should say so. But I believe 
trying to guilt others into writing the needed tests is neither fair 
nor effective in the long term.

Snarking publicly might make you feel better, but it makes for an 
unpleasant community and is likely to alienate more people than it will 
inspire.

> And once again, if some stupid idiot
> volunteer bothers to put in a patch to the code and/or the unit test,
> and it sits and rots for five years, is that person likely to come
> back to Python?  Strangely, some do.

I'm not sure that describing people who provide patches as "stupid idiot 
volunteers" is a good way to motivate more people to provide patches. 
But either way, I think you answer your own question: apparently some 
people *are* understanding of the reasons that issues sometimes get 
missed or neglected.



-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com