Re: [Python-Dev] AST optimizer implemented in Python

2012-08-13 Thread Mark Shannon 

Brett Cannon wrote:



On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow > wrote:


On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon mailto:[email protected]>> wrote:
 > It would also be very easy to expand importlib.abc.SourceLoader
to add a
 > method which is called with source and returns the bytecode to be
written
 > out

Yes, please.  Not having to hack around this would be nice.


http://bugs.python.org/issue15627


AST transformation is a lot more general than just optimization.

Adding an AST transformation is a relatively painless way to add to
Python the last element of lisp-ness that it lacks:
Namely being able to treat code as data and transform it at runtime,
after parsing but before execution.

Some examples:
Profiling code be added by an AST transformation.
IMO this would have been a more elegant way to implement CProfile
and similar profilers than the current approach.

AST transformations allow DSLs to be implemented in Python
(I don't know if that is a + or - ).

Access to the AST of a function at runtime would also be of use to
method-based dynamic optimizers, or dynamic de-optimizers for static
compilers.

All for the price of adding a single method to SourceLoader.
What a bargain :)

Cheers,
Mark.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New in Python 3.3

2012-08-13 Thread Matthias Klose 
On 09.08.2012 01:04, Victor Stinner wrote:
> Does Python 3.3 support cross-compilation? There are two new option
> for configure: --host and --build, but it's not mentioned in What's
> New in Python 3.3.

it does work, but it is only tested for the linux -> linux case. the mingw and
macosx cross builds did require changes, which didn't go into 3.3 before the
first beta release. what is completely missing is the cross build infrastructure
to build third party extensions. so maybe it's a bit early to announce it in the
release notes.

  Matthias


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 2.7 + https + urlopen = ?

2012-08-13 Thread martin 

How would you feel, if you issued :

import urllib
urlopen("""https://server.domain.com""";).read()

and the command got you data from some other URL without telling  
you! You use firefox, and the site is different than the data you  
got! Same with chrome. Safari. Even IE !

Cheated? (Well I was mad -- after IE worked).

[...]
None of them worked! Wow. Then you wonder, whats going on. You poke  
one of the server administrator, and he sends you the logs, and you  
see the problem. The keyword being "SNI".


I believe there is a bug in the HTTP server; it doesn't conform to the
HTTP/1.1 protocol. Even without the client using SNI, you should still
get the right page, since the HTTP Host: header indicates the host you
are trying to contact at this point, not SNI. The SNI is only relevant
for the certificate that the server presents.

Regards,
Martin


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST optimizer implemented in Python

2012-08-13 Thread Guido van Rossum 
Not so fast. If you make this a language feature you force all Python
implementations to support an identical AST API. That's a big step.

Not that AST manipulation isn't cool -- but I'd like to warn against
over-enthusiasm that might backfire on the language (or its community)
as a whole.

--Guido

On Mon, Aug 13, 2012 at 1:34 AM, Mark Shannon  wrote:
> Brett Cannon wrote:
>>
>>
>>
>> On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow > > wrote:
>>
>> On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon > > wrote:
>>  > It would also be very easy to expand importlib.abc.SourceLoader
>> to add a
>>  > method which is called with source and returns the bytecode to be
>> written
>>  > out
>>
>> Yes, please.  Not having to hack around this would be nice.
>>
>>
>> http://bugs.python.org/issue15627
>
>
> AST transformation is a lot more general than just optimization.
>
> Adding an AST transformation is a relatively painless way to add to
> Python the last element of lisp-ness that it lacks:
> Namely being able to treat code as data and transform it at runtime,
> after parsing but before execution.
>
> Some examples:
> Profiling code be added by an AST transformation.
> IMO this would have been a more elegant way to implement CProfile
> and similar profilers than the current approach.
>
> AST transformations allow DSLs to be implemented in Python
> (I don't know if that is a + or - ).
>
> Access to the AST of a function at runtime would also be of use to
> method-based dynamic optimizers, or dynamic de-optimizers for static
> compilers.
>
> All for the price of adding a single method to SourceLoader.
> What a bargain :)
>
> Cheers,
> Mark.
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST optimizer implemented in Python

2012-08-13 Thread Terry Reedy 

On 8/13/2012 10:45 AM, Guido van Rossum wrote:

Not so fast. If you make this a language feature you force all Python
implementations to support an identical AST API. That's a big step.


I have been wondering about this. One could think from the manuals that 
we are there already. From the beginning of the ast chapter:


"The ast module helps Python applications to process trees of *the* 
Python abstract syntax grammar.  ... An abstract syntax tree can be 
generated by passing ast.PyCF_ONLY_AST as a flag to the compile() 
built-in function" (emphasis on *the* added).


and the entry for compile(): "Compile the source into a code or AST object."

I see nothing about ast possibly being CPython only. Should there be?

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Brett Cannon 
On Mon, Aug 13, 2012 at 3:00 PM, Terry Reedy  wrote:

> On 8/13/2012 10:45 AM, Guido van Rossum wrote:
>
>> Not so fast. If you make this a language feature you force all Python
>> implementations to support an identical AST API. That's a big step.
>>
>
> I have been wondering about this. One could think from the manuals that we
> are there already. From the beginning of the ast chapter:
>
> "The ast module helps Python applications to process trees of *the* Python
> abstract syntax grammar.  ... An abstract syntax tree can be generated by
> passing ast.PyCF_ONLY_AST as a flag to the compile() built-in function"
> (emphasis on *the* added).
>
> and the entry for compile(): "Compile the source into a code or AST
> object."
>
> I see nothing about ast possibly being CPython only. Should there be?


Time to ask the other VMs what they are currently doing (the ast module
came into existence in Python 2.6 so all the VMs should be answer the
question since Jython is in alpha for 2.7 compatibility).
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Push importlib ABC hierarchy chart.

2012-08-13 Thread Brett Cannon 
For documentation, this appears to be for http://bugs.python.org/issue15628
 .

On Mon, Aug 13, 2012 at 3:19 PM, andrew.svetlov
wrote:

> http://hg.python.org/cpython/rev/1c8a6df94602
> changeset:   78547:1c8a6df94602
> user:Andrew Svetlov 
> date:Mon Aug 13 22:19:01 2012 +0300
> summary:
>   Push importlib ABC hierarchy chart.
>
> files:
>   Doc/library/importlib.rst |  15 +++
>   1 files changed, 15 insertions(+), 0 deletions(-)
>
>
> diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
> --- a/Doc/library/importlib.rst
> +++ b/Doc/library/importlib.rst
> @@ -121,6 +121,21 @@
>  used by :keyword:`import`. Some subclasses of the core abstract base
> classes
>  are also provided to help in implementing the core ABCs.
>
> +ABC hierarchy::
> +
> +object
> + +-- Finder
> + |+-- MetaPathFinder
> + |+-- PathEntryFinder
> + +-- Loader
> +  +-- ResourceLoader +
> +  +-- InspectLoader  |
> +   +-- ExecutionLoader --+
> + +-- FileLoader
> + +-- SourceLoader
> +  +-- PyLoader
> +  +-- PyPycLoader
> +
>
>  .. class:: Finder
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread [email protected] 
On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon  wrote:
>> I see nothing about ast possibly being CPython only. Should there be?
>
>
> Time to ask the other VMs what they are currently doing (the ast module came
> into existence in Python 2.6 so all the VMs should be answer the question
> since Jython is in alpha for 2.7 compatibility).
2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
ast.py. I haven't applied the same obsessiveness to 2.7, but I do
intend to look closely at Jython's ast.py results compared to
CPython's in the 3.x effort. Also I plan to allow some backwards
compatibility compromises between early point releases of our 2.7
series, as I want to apply what I learn in our 3.x effort to 2.7 point
releases, so we should be able to keep up with most simple ast.py
changes. I'm not so sure that the current discussion are going to be
"simple though" :) -- if it's pure python we should hopefully be
alright.

-Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread [email protected] 
On Mon, Aug 13, 2012 at 1:05 PM, [email protected]
 wrote:
> 2.5+ contains
I should have said *Jython* 2.5+

-Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Guido van Rossum 
On Mon, Aug 13, 2012 at 1:05 PM, [email protected]
 wrote:
> On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon  wrote:
>>> I see nothing about ast possibly being CPython only. Should there be?
>>
>>
>> Time to ask the other VMs what they are currently doing (the ast module came
>> into existence in Python 2.6 so all the VMs should be answer the question
>> since Jython is in alpha for 2.7 compatibility).

[Jython]
> 2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
> ast.py.

But CPython's ast.py contains very little code -- it's all done in ast.c.

Still, I'm glad you are actually considering this a cross-language
feature, and I will gladly retract my warning. (Still, I don't know if
it is subject to the usual backward compatibility constraints.)

> I haven't applied the same obsessiveness to 2.7, but I do
> intend to look closely at Jython's ast.py results compared to
> CPython's in the 3.x effort. Also I plan to allow some backwards
> compatibility compromises between early point releases of our 2.7
> series, as I want to apply what I learn in our 3.x effort to 2.7 point
> releases, so we should be able to keep up with most simple ast.py
> changes. I'm not so sure that the current discussion are going to be
> "simple though" :) -- if it's pure python we should hopefully be
> alright.

It might be pure python for Jython, but it's not for CPython.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread [email protected] 
On Mon, Aug 13, 2012 at 1:46 PM, Guido van Rossum  wrote:
> On Mon, Aug 13, 2012 at 1:05 PM, [email protected]
>  wrote:
>> On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon  wrote:
 I see nothing about ast possibly being CPython only. Should there be?
>>>
>>>
>>> Time to ask the other VMs what they are currently doing (the ast module came
>>> into existence in Python 2.6 so all the VMs should be answer the question
>>> since Jython is in alpha for 2.7 compatibility).
>
> [Jython]
>> 2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
>> ast.py.
>
> But CPython's ast.py contains very little code -- it's all done in ast.c.
What I did was dump a pretty print of the ast from Jython and CPython
for every file in Lib/* and diff the results with a script. I got the
differences down to a small number of minor variations.

> Still, I'm glad you are actually considering this a cross-language
> feature, and I will gladly retract my warning. (Still, I don't know if
> it is subject to the usual backward compatibility constraints.)
I don't know if IronPython does the same though... we might want to
wait for them to respond.

> It might be pure python for Jython, but it's not for CPython.
It's actually Java for us :) -- in fact the internal AST uses the
exact Java that is exposed from our _ast.py - which I've come to
regard as a mistake (though it was useful at the time). I want to do
the same obsessive diff game with 3.x but then probably separate out
our internal ast implementation (possibly making ast.py pure Python).

BTW - is Python's internal AST exactly exposed by ast.py or is there a
separate internal AST implementation?

-Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Terry Reedy 

On 8/13/2012 4:46 PM, Guido van Rossum wrote:

On Mon, Aug 13, 2012 at 1:05 PM,[email protected]
  wrote:

>On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon  wrote:

>>>I see nothing about ast possibly being CPython only. Should there be?

>>
>>
>>Time to ask the other VMs what they are currently doing (the ast module came
>>into existence in Python 2.6 so all the VMs should be answer the question
>>since Jython is in alpha for 2.7 compatibility).

[Jython]

>2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
>ast.py.

But CPython's ast.py contains very little code -- it's all done in ast.c.

Still, I'm glad you are actually considering this a cross-language
feature, and I will gladly retract my warning. (Still, I don't know if
it is subject to the usual backward compatibility constraints.)


I should have quoted a bit more. After the first sentence
"The ast module helps Python applications to process trees of the Python 
abstract syntax grammar."

the next sentence is
"The abstract syntax itself might change with each Python release; this 
module helps to find out programmatically what the current grammar looks 
like."


The 'current grammar' is given in 30.2.2. Abstract Grammar.

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Brett Cannon 
On Aug 13, 2012 5:22 PM, "[email protected]" 
wrote:
>
> On Mon, Aug 13, 2012 at 1:46 PM, Guido van Rossum 
wrote:
> > On Mon, Aug 13, 2012 at 1:05 PM, [email protected]
> >  wrote:
> >> On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon 
wrote:
>  I see nothing about ast possibly being CPython only. Should there be?
> >>>
> >>>
> >>> Time to ask the other VMs what they are currently doing (the ast
module came
> >>> into existence in Python 2.6 so all the VMs should be answer the
question
> >>> since Jython is in alpha for 2.7 compatibility).
> >
> > [Jython]
> >> 2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
> >> ast.py.
> >
> > But CPython's ast.py contains very little code -- it's all done in
ast.c.
> What I did was dump a pretty print of the ast from Jython and CPython
> for every file in Lib/* and diff the results with a script. I got the
> differences down to a small number of minor variations.
>
> > Still, I'm glad you are actually considering this a cross-language
> > feature, and I will gladly retract my warning. (Still, I don't know if
> > it is subject to the usual backward compatibility constraints.)
> I don't know if IronPython does the same though... we might want to
> wait for them to respond.
>
> > It might be pure python for Jython, but it's not for CPython.
> It's actually Java for us :) -- in fact the internal AST uses the
> exact Java that is exposed from our _ast.py - which I've come to
> regard as a mistake (though it was useful at the time). I want to do
> the same obsessive diff game with 3.x but then probably separate out
> our internal ast implementation (possibly making ast.py pure Python).
>
> BTW - is Python's internal AST exactly exposed by ast.py or is there a
> separate internal AST implementation?

Direct. There is an AST grammar file that gets compiled into C and Python
objects which are used by the compiler (c version) or exposed to users
(Python version).

>
> -Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default): Merge 3.2

2012-08-13 Thread Brian Curtin 
On Mon, Aug 13, 2012 at 5:13 PM, brian.curtin
 wrote:
> http://hg.python.org/cpython/rev/256bfee696c5
> changeset:   78552:256bfee696c5
> parent:  78549:edcbf3edf701
> parent:  78551:fcad4566910b
> user:Brian Curtin 
> date:Mon Aug 13 17:12:02 2012 -0500
> summary:
>   Merge 3.2
>
> files:
>   Lib/test/support.py |   68 +++-
>   Misc/NEWS   |  265 

this Misc/NEWS disaster didn't appear in the diff. Fixing now...
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] AST optimizer implemented in Python

2012-08-13 Thread Nick Coghlan 
Implementations are currently required to *have* an AST (or else declare
non compliance with that particular flag to compile). They're definitely
not required to have the *same* AST, thus all AST manipulation, like
bytecode manipulation, is necessarily implementation dependent.

We don't even guarantee AST compatibility between versions of CPython.

I believe the appropriate warnings are already present in the ast module
docs, but there may be misleading wording elsewhere that needs to be
cleaned up.

Regards,
Nick.

--
Sent from my phone, thus the relative brevity :)
On Aug 14, 2012 5:03 AM, "Terry Reedy"  wrote:

> On 8/13/2012 10:45 AM, Guido van Rossum wrote:
>
>> Not so fast. If you make this a language feature you force all Python
>> implementations to support an identical AST API. That's a big step.
>>
>
> I have been wondering about this. One could think from the manuals that we
> are there already. From the beginning of the ast chapter:
>
> "The ast module helps Python applications to process trees of *the* Python
> abstract syntax grammar.  ... An abstract syntax tree can be generated by
> passing ast.PyCF_ONLY_AST as a flag to the compile() built-in function"
> (emphasis on *the* added).
>
> and the entry for compile(): "Compile the source into a code or AST
> object."
>
> I see nothing about ast possibly being CPython only. Should there be?
>
> --
> Terry Jan Reedy
>
> __**_
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/**mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> ncoghlan%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread [email protected] 
On Mon, Aug 13, 2012 at 3:10 PM, Brett Cannon  wrote:

> Direct. There is an AST grammar file that gets compiled into C and Python
> objects which are used by the compiler (c version) or exposed to users
> (Python version).
At the risk of making you repeat yourself, and just to be sure I
understand: There are C objects used by the compiler and Python
objects that are exposed to the users (written in C though) that are
generated by the AST grammar. That at least sounds like they are
different. The last I checked the grammar was Python.asdl and the
translater was asdl_c.py resulting in /Python/Python-ast.c which looks
like it is the implementation of _ast.py

Are the AST objects from Python-ast.c used by the compiler? And what
is the relationship between Python-ast.c and /Python/ast.c? And what
about the CST mentioned at the top of /Python/ast.c?

I ask all of this because I want to be sure that separating the
internal AST in Jython from the one exposed in ast.py is really a good
idea. If CPython does not make this distinction that will be a strike
against the idea.

-Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Brett Cannon 
On Mon, Aug 13, 2012 at 6:35 PM, [email protected] <
[email protected]> wrote:

> On Mon, Aug 13, 2012 at 3:10 PM, Brett Cannon  wrote:
>
> > Direct. There is an AST grammar file that gets compiled into C and Python
> > objects which are used by the compiler (c version) or exposed to users
> > (Python version).
> At the risk of making you repeat yourself, and just to be sure I
> understand: There are C objects used by the compiler and Python
> objects that are exposed to the users (written in C though) that are
> generated by the AST grammar.


Both sets of objects are generated from the grammar. It's wrapping some C
structs (the C version of the AST) in an extension module where the fields
of the struct and the names of the types are all the same (the Python
version) no matter if it is C or Python. Converting between the two is just
a matter of allocating memory and copying data from one struct to another.


> That at least sounds like they are
> different.


Are you asking if we pass the objects through transparently, or if they are
just the same API? The are the same API since the AST nodes used by the
compiler just have an extension exposing them that has the same names,
fields, etc. But to expose the API to Python code the C-level objects are
taken, pulled apart, and used to populate and exact API copy of them as
Python object (i.e. the ast2obj_* functions defined in Python/Python-ast.c).

To try to make this really clear, consider the Assign node type. At the C
level it's just a struct::

struct {
asdl_seq *targets;
expr_ty value;
} Assign;

An asdl_seq is just an array of AST nodes. So converting to Python code is
just a matter of allocating the equivalent Assign_type (which is a
PythonTypeObject), and then populating its 'targets' attribute with a list
of the AST nodes and its expr instance for its 'value' type. It's all very
mechanical since it is all code-generated.


> The last I checked the grammar was Python.asdl and the
> translater was asdl_c.py resulting in /Python/Python-ast.c which looks
> like it is the implementation of _ast.py
>

There is no _ast.py, only Lib/ast.py which just provides helper code for
working with the AST (e.g. a NodeVisitor class). The builtin _ast module
comes from Python/Python-ast.c.


>
> Are the AST objects from Python-ast.c used by the compiler? And what
> is the relationship between Python-ast.c and /Python/ast.c? And what
> about the CST mentioned at the top of /Python/ast.c?
>

http://docs.python.org/devguide/compiler.html explains it all.


>
> I ask all of this because I want to be sure that separating the
> internal AST in Jython from the one exposed in ast.py is really a good
> idea. If CPython does not make this distinction that will be a strike
> against the idea.
>

As I said, depends if you mean API or actual objects. The compiler itself
uses C objects which are nothing more than structs and unions. The AST
exposed by the _ast module uses the same names, fields, etc., but are
actual Python objects instead of structs and unions. The separation allows
the compiler to save on memory costs by only using structs instead of a
complete PyObject struct which would have tons of stuff that the compiler
doesn't need (e.g. the AST has no methods so why waste memory on PyObject
allocation for method slots that will never be set?).
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Steven D'Aprano 

On 14/08/12 06:46, Guido van Rossum wrote:

On Mon, Aug 13, 2012 at 1:05 PM, [email protected]
  wrote:

On Mon, Aug 13, 2012 at 12:06 PM, Brett Cannon  wrote:

I see nothing about ast possibly being CPython only. Should there be?



Time to ask the other VMs what they are currently doing (the ast module came
into existence in Python 2.6 so all the VMs should be answer the question
since Jython is in alpha for 2.7 compatibility).


[Jython]

2.5+ contains an ast.py that I obsessively compared to CPython's 2.5
ast.py.


But CPython's ast.py contains very little code -- it's all done in ast.c.

Still, I'm glad you are actually considering this a cross-language
feature, and I will gladly retract my warning. (Still, I don't know if
it is subject to the usual backward compatibility constraints.)


Well, that's Jython. What about IronPython, TinyPy, CLPython, etc. and
future implementations?

Perhaps ast should be considered a quality of implementation module.
Lack of one does not disqualify from being "Python", but it does make it
a second-class implementation.




--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread [email protected] 
Thanks Brett, that cleared everything up for me! And indeed it is what
I'm thinking of doing for Jython (Minimal nodes for the compiler and
parallel PyObjects for Python).

-Frank
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] do all VMs implement the ast module? (was: Re: AST optimizer implemented in Python)

2012-08-13 Thread Alex Gaynor 
Brett Cannon  python.org> writes:

> 
> Time to ask the other VMs what they are currently doing (the ast module came 
into existence in Python 2.6 so all the VMs should be answer the question since 
Jython is in alpha for 2.7 compatibility).
> 

As far as I know PyPy supports the ast module, and produces ASTs that are the
same as CPython's. That said I do regard this as an implementation detail,
further I'm guessing this is the context of the AST optimizer thread, and though
I have neither the time nor the inclination to wade into that, put me down as -1
a) everything proposed there is possible, b) making this a front-and-center API
makes it really easy to shoot themselves in the foot, by doing things like
breaking Python with invalid optimizations (hint: almost every optimization
proposed in that thread is invalid in the general case).

Alex


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com