Re: [Python-Dev] [Python-checkins] cpython: Issue #15502: Finish bringing importlib.abc in line with the current
On Fri, 10 Aug 2012 16:56:43 -0700, Chris Jerdonek wrote: > On Fri, Aug 10, 2012 at 9:21 AM, brett.cannon > wrote: > > http://hg.python.org/cpython/rev/0a75ce232f56 > > changeset: 78485:0a75ce232f56 > > user:Brett Cannon > > date:Fri Aug 10 12:21:12 2012 -0400 > > summary: > > Issue #15502: Finish bringing importlib.abc in line with the current > > > + cache used by the finder. Used by :func:`invalidate_caches()` when > > Minor style nit: the Dev Guide says not to include the trailing > parentheses in :func: text: > > "func: The name of a Python function; dotted names may be used. The > role text should not include trailing parentheses to enhance > readability..." > > (from http://hg.python.org/devguide/file/f518f23d06d5/documenting.rst#l888 ) > > (though I don't know why the Dev Guide says the opposite for :c:func: > and is silent on :meth:.) To clarify: :func: automatically adds the ()s, so if you put them in the source you get double: invalidate_caches()(). As Chris said, use the 'alternate text' form if you want to show a call with arguments. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #15502: Finish bringing importlib.abc in line with the current
On 08/11/2012 04:49 PM, R. David Murray wrote: > On Fri, 10 Aug 2012 16:56:43 -0700, Chris Jerdonek > wrote: >> On Fri, Aug 10, 2012 at 9:21 AM, brett.cannon >> wrote: >> > http://hg.python.org/cpython/rev/0a75ce232f56 >> > changeset: 78485:0a75ce232f56 >> > user:Brett Cannon >> > date:Fri Aug 10 12:21:12 2012 -0400 >> > summary: >> > Issue #15502: Finish bringing importlib.abc in line with the current >> >> > + cache used by the finder. Used by :func:`invalidate_caches()` when >> >> Minor style nit: the Dev Guide says not to include the trailing >> parentheses in :func: text: >> >> "func: The name of a Python function; dotted names may be used. The >> role text should not include trailing parentheses to enhance >> readability..." >> >> (from http://hg.python.org/devguide/file/f518f23d06d5/documenting.rst#l888 ) >> >> (though I don't know why the Dev Guide says the opposite for :c:func: >> and is silent on :meth:.) > > To clarify: :func: automatically adds the ()s, so if you put them in > the source you get double: invalidate_caches()(). That is not true: they are stripped if present before they are added again. What will give double parens is if you don't leave them empty, such as :func:`invalidate_caches(foo)`. This is because the reference markup is not meant for code snippets. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #15502: Finish bringing importlib.abc in line with the current
On Sat, Aug 11, 2012 at 9:40 AM, Georg Brandl wrote: > On 08/11/2012 04:49 PM, R. David Murray wrote: >> On Fri, 10 Aug 2012 16:56:43 -0700, Chris Jerdonek >> wrote: >>> On Fri, Aug 10, 2012 at 9:21 AM, brett.cannon >>> wrote: >>> > http://hg.python.org/cpython/rev/0a75ce232f56 >>> > changeset: 78485:0a75ce232f56 >>> > user:Brett Cannon >>> > date:Fri Aug 10 12:21:12 2012 -0400 >>> > summary: >>> > Issue #15502: Finish bringing importlib.abc in line with the current >>> >>> > + cache used by the finder. Used by :func:`invalidate_caches()` when >>> >>> Minor style nit: the Dev Guide says not to include the trailing >>> parentheses in :func: text: >>> >>> "func: The name of a Python function; dotted names may be used. The >>> role text should not include trailing parentheses to enhance >>> readability..." >>> >>> (from http://hg.python.org/devguide/file/f518f23d06d5/documenting.rst#l888 ) >>> >>> (though I don't know why the Dev Guide says the opposite for :c:func: >>> and is silent on :meth:.) >> >> To clarify: :func: automatically adds the ()s, so if you put them in >> the source you get double: invalidate_caches()(). > > That is not true: they are stripped if present before they are added again. Yes, and the Dev Guide in part says this afterward. Its statement not to include trailing parentheses to enhance readability was I assume for readability of the .rst files rather than readability of the output. --Chris > What will give double parens is if you don't leave them empty, such as > :func:`invalidate_caches(foo)`. This is because the reference markup is not > meant for code snippets. > > Georg > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] AST optimizer implemented in Python
Hi, I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project. https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py To test its peephole optimizations (by checking manually its final bytecode), I wrote a patch for Python to disable Python internal peephole optimizer (on bytecode): https://bitbucket.org/haypo/misc/src/tip/python/compile_disable_peephole.patch -- There is BytecodeAssembler [1], but it seems to be specialized on bytecode. There are (at least?) 3 different issues to implement an AST optimizer, but in C, not in Python: http://bugs.python.org/issue1346238 http://bugs.python.org/issue10399 http://bugs.python.org/issue11549 -- My proof-of-concept only implements very basic optimizations like 1+1 => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do more interesting optimization like function inlining. To allow more optimization, the optimizer permits to declare variables as constant. For example, sys.hexversion is replaced by its value by the optimizer, and checks like "sys.hexversion > 0x300" are done at compile time. This feature can be used to drop completly debug code at compilation, without the need of a preprocessor. For example, calls to logging.debug() can simply be dropped if the log level is hardcoded to ERROR (40). Other idea to improve this optimizer: - move invariant out of loops. Example: "x=[]; for i in range(10): x.append(i)" => "x=[]; x_append=x.append; for i in range(10): x_append(i)". Require to infer the type of variables. - unroll loops - inline small functions - etc. To be able to use such optimizer on a whole Python project, a new function (ex: sys.setastoptimizer()) can be added to CPython to call the optimizer between the compilation to AST and the compilation to bytecode. So the optimizer would be optionnal and it avoids bootstrap issues. [1] http://pypi.python.org/pypi/BytecodeAssembler Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 11:30 AM, Victor Stinner wrote: > Hi, > > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py Neat! > > > To test its peephole optimizations (by checking manually its final > bytecode), I wrote a patch for Python to disable Python internal > peephole optimizer (on bytecode): > > https://bitbucket.org/haypo/misc/src/tip/python/compile_disable_peephole.patch > > -- > > There is BytecodeAssembler [1], but it seems to be specialized on > bytecode. There are (at least?) 3 different issues to implement an AST > optimizer, but in C, not in Python: > > http://bugs.python.org/issue1346238 > http://bugs.python.org/issue10399 > http://bugs.python.org/issue11549 > > -- > > My proof-of-concept only implements very basic optimizations like 1+1 > => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do > more interesting optimization like function inlining. > > To allow more optimization, the optimizer permits to declare variables > as constant. For example, sys.hexversion is replaced by its value by > the optimizer, and checks like "sys.hexversion > 0x300" are done > at compile time. This feature can be used to drop completly debug code > at compilation, without the need of a preprocessor. For example, calls > to logging.debug() can simply be dropped if the log level is hardcoded > to ERROR (40). > > Other idea to improve this optimizer: > - move invariant out of loops. Example: "x=[]; for i in range(10): > x.append(i)" => "x=[]; x_append=x.append; for i in range(10): > x_append(i)". Require to infer the type of variables. > - unroll loops > - inline small functions > - etc. > > To be able to use such optimizer on a whole Python project, a new > function (ex: sys.setastoptimizer()) can be added to CPython to call > the optimizer between the compilation to AST and the compilation to > bytecode. So the optimizer would be optionnal and it avoids bootstrap > issues. > +1 We should add some form of setastoptimizer API in 3.4. Please start a PEP for this. It would be nice to include the ability to properly cache the ast optimizer output so that it does not have to run every time (in pyc files or similar, etc) but can verify that it is the specific ast optimizer that ran on a give existing cached copy. -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 11:47 AM, Gregory P. Smith wrote: > > On Sat, Aug 11, 2012 at 11:30 AM, Victor Stinner > wrote: >> >> Hi, >> >> I started to implement an AST optimizer in Python. It's easy to create >> a new AST tree, so I'm surprised that I didn't find any existing >> project. >> >> https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py > > > Neat! > > +1 We should add some form of setastoptimizer API in 3.4. Please start a > PEP for this. It would be nice to include the ability to properly cache the > ast optimizer output so that it does not have to run every time (in pyc > files or similar, etc) but can verify that it is the specific ast optimizer > that ran on a give existing cached copy. Once .pyc is created, I do not think that we keep the AST around any longer. Otherwise, I wouldn't have to write PyXfuscator. https://bitbucket.org/namn/pyxfuscator Or perhaps I am misunderstanding you. Nam ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
+1 We should add some form of setastoptimizer API in 3.4. Please start a PEP for this. It would be nice to include the ability to properly cache the ast optimizer output so that it does not have to run every time (in pyc files or similar, etc) but can verify that it is the specific ast optimizer that ran on a give existing cached copy. Once .pyc is created, I do not think that we keep the AST around any longer. Otherwise, I wouldn't have to write PyXfuscator. https://bitbucket.org/namn/pyxfuscator Or perhaps I am misunderstanding you. I think you misunderstood. What gps is concerned about (IIUC) that some people add ast optimizers in some run of Python, but other AST optimizers in a different run. Then, if you use a Python byte code file, you should be able to find out what AST optimizers have been run to create the pyc file, so you know whether you have to recompile or not. Of course, if that is really a desirable feature, you may want multiple pyc files, per combination of AST optimizers. The __pycache__ directory would readily support that. This seems to get complicated quickly, so a PEP is indeed desirable. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On 8/11/2012 2:30 PM, Victor Stinner wrote: Hi, I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project. https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py To test its peephole optimizations (by checking manually its final bytecode), I wrote a patch for Python to disable Python internal peephole optimizer (on bytecode): https://bitbucket.org/haypo/misc/src/tip/python/compile_disable_peephole.patch I would very much like to see the ability to disable all optimizers. As work continues on the various forms of optimization, please remember that sometimes programs are executed to reason about them, for example, when measuring test coverage, or when debugging. Just as gcc has a -O0 option to disable all code optimizations so that you can more easily debug programs, it would be fabulous for CPython to have an option to disable all code optimizations. This would include the existing peephole optimizer as well as any new optimizers such as Victor's AST optimizer. Full disclosure: I previously raised this possibility in ticket, and it was not as popular as I had hoped: http://bugs.python.org/issue2506 --Ned. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 3:27 PM, "Martin v. Löwis" wrote: > I think you misunderstood. What gps is concerned about (IIUC) that some > people add ast optimizers in some run of Python, but other AST optimizers in > a different run. Then, if you use a Python byte code > file, you should be able to find out what AST optimizers have been > run to create the pyc file, so you know whether you have to recompile > or not. > > Of course, if that is really a desirable feature, you may want multiple > pyc files, per combination of AST optimizers. The __pycache__ directory > would readily support that. Perhaps it's a bit of an abuse, but modifying sys.implementation.cache_tag should result in separate .pyc file in __pycache__. Using the same cache tag later would mean that .pyc file gets loaded during import. -eric ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sun, Aug 12, 2012 at 4:30 AM, Victor Stinner wrote: > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. Very nice idea! > Other idea to improve this optimizer: > - move invariant out of loops. Example: "x=[]; for i in range(10): > x.append(i)" => "x=[]; x_append=x.append; for i in range(10): > x_append(i)". Require to infer the type of variables. But this is risky. It's theoretically possible for x.append to replace itself. Sure it may not be a normal or common thing to do, but it's possible. And yet it could be pretty advantageous for most programs. Perhaps the best way is to hide potentially-risky optimizations behind command-line options? The default mode could be to do every change that's guaranteed not to affect execution, and everything else is an extra (like French, music, and washing). ChrisA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 2:30 PM, Victor Stinner wrote: > Hi, > > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py > > To test its peephole optimizations (by checking manually its final > bytecode), I wrote a patch for Python to disable Python internal > peephole optimizer (on bytecode): > > https://bitbucket.org/haypo/misc/src/tip/python/compile_disable_peephole.patch > > -- > > There is BytecodeAssembler [1], but it seems to be specialized on > bytecode. There are (at least?) 3 different issues to implement an AST > optimizer, but in C, not in Python: > > http://bugs.python.org/issue1346238 > http://bugs.python.org/issue10399 > http://bugs.python.org/issue11549 > > -- > > My proof-of-concept only implements very basic optimizations like 1+1 > => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do > more interesting optimization like function inlining. > > To allow more optimization, the optimizer permits to declare variables > as constant. For example, sys.hexversion is replaced by its value by > the optimizer, and checks like "sys.hexversion > 0x300" are done > at compile time. This feature can be used to drop completly debug code > at compilation, without the need of a preprocessor. For example, calls > to logging.debug() can simply be dropped if the log level is hardcoded > to ERROR (40). > > Other idea to improve this optimizer: > - move invariant out of loops. Example: "x=[]; for i in range(10): > x.append(i)" => "x=[]; x_append=x.append; for i in range(10): > x_append(i)". Require to infer the type of variables. > - unroll loops > - inline small functions > - etc. > > To be able to use such optimizer on a whole Python project, a new > function (ex: sys.setastoptimizer()) can be added to CPython to call > the optimizer between the compilation to AST and the compilation to > bytecode. So the optimizer would be optionnal and it avoids bootstrap > issues. > It would also be very easy to expand importlib.abc.SourceLoader to add a method which is called with source and returns the bytecode to be written out which people could override with AST optimizations before sending the bytecode back. That way we don't have to get into the whole business of AST transformations if we don't want to (although, as Victor pointed out, there are some people who do want this formally supported). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon wrote: > It would also be very easy to expand importlib.abc.SourceLoader to add a > method which is called with source and returns the bytecode to be written > out Yes, please. Not having to hack around this would be nice. > which people could override with AST optimizations before sending the > bytecode back. That way we don't have to get into the whole business of AST > transformations if we don't want to (although, as Victor pointed out, there > are some people who do want this formally supported). Also cool. -eric ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow wrote: > On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon wrote: > > It would also be very easy to expand importlib.abc.SourceLoader to add a > > method which is called with source and returns the bytecode to be written > > out > > Yes, please. Not having to hack around this would be nice. > http://bugs.python.org/issue15627 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
>> Other idea to improve this optimizer:
>> - move invariant out of loops. Example: "x=[]; for i in range(10):
>> x.append(i)" => "x=[]; x_append=x.append; for i in range(10):
>> x_append(i)". Require to infer the type of variables.
>
> But this is risky. It's theoretically possible for x.append to replace
> itself.
For this specific example, x.append cannot be modified: it raises
AttributeError('list' object attribute 'append' is read-only).
The idea would be to allow the developer to specify explicitly what he
wants to optimize. I'm using a configuration class with a list of what
can be optimized (ex: len(int)), but it can be changed to something
different later.
It must be configurable to be able to specify: "this specific variable
is constant in my project"..
> Perhaps the best way is to hide potentially-risky optimizations behind
> command-line options? The default mode could be to do every change
> that's guaranteed not to affect execution, and everything else is an
> extra (like French, music, and washing).
I don't care of integration into Python yet (but it would be nice to
prepare such feature in Python 3.4). It can be a third party module,
something like:
import ast_optimizer
ast_optimizer.hack_import_machinery()
ast_optimizer.constants.add('application.DEBUG')
(added on the top of your main script)
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
On Sun, Aug 12, 2012 at 11:17 AM, Victor Stinner wrote: > The idea would be to allow the developer to specify explicitly what he > wants to optimize. I'm using a configuration class with a list of what > can be optimized (ex: len(int)), but it can be changed to something > different later. > > It must be configurable to be able to specify: "this specific variable > is constant in my project".. That sounds like a plan. It's possibly even worth making blanket statements like "in this module, once a name is bound to an object, it will never be rebound to an object of different type". ChrisA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
Chris Angelico, 12.08.2012 01:22: >> Other idea to improve this optimizer: >> - move invariant out of loops. Example: "x=[]; for i in range(10): >> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >> x_append(i)". Require to infer the type of variables. > > But this is risky. It's theoretically possible for x.append to replace > itself. Sure it may not be a normal or common thing to do, but it's > possible. Not only that. It changes semantics. If x.append is not defined, the exception would now be raised outside of the loop, and the loop itself may have side-effects already. In fact, the mere lookup of x.append may have side effects as well ... Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
Stefan Behnel, 12.08.2012 06:42: > Chris Angelico, 12.08.2012 01:22: >>> Other idea to improve this optimizer: >>> - move invariant out of loops. Example: "x=[]; for i in range(10): >>> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >>> x_append(i)". Require to infer the type of variables. >> >> But this is risky. It's theoretically possible for x.append to replace >> itself. Sure it may not be a normal or common thing to do, but it's >> possible. > > Not only that. It changes semantics. If x.append is not defined, the > exception would now be raised outside of the loop, and the loop itself may > have side-effects already. In fact, the mere lookup of x.append may have > side effects as well ... That being said, the specific case above can be optimised, we do something like this in Cython, too. It requires both (simple) type inference and control flow analysis, though, because you need to know that a) x holds a list and b) it was assigned outside of the loop and is not being assigned to inside. So it might look simple and obvious, but it requires quite a bit of compiler infrastructure. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
Victor Stinner, 11.08.2012 20:30:
> I started to implement an AST optimizer in Python. It's easy to create
> a new AST tree, so I'm surprised that I didn't find any existing
> project.
>
> https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py
Since you're about to do pretty much the same thing, here is the code that
Cython uses for its static tree optimisations. It's not based on Python's
own AST, but most of the concepts should apply more or less directly to
that as well.
https://github.com/cython/cython/blob/master/Cython/Compiler/Optimize.py
The code uses a visitor pattern, so you basically register a tree node
callback method by naming it after the node class, e.g. "visit_NameNode()"
to intercept on variables etc. For processing builtins, we use the same
mechanism with method names like "_handle_simple_method_bytes_decode()" for
a ".decode()" method call on a "bytes" object that receives no keyword
arguments or args/kwargs ("simple").
Constant folding is a bit funny in that it calculates lots of constants,
but not all of them will be used in the code. Some are just kept around in
tree nodes to serve as support for later optimisations. Very helpful.
Some more tree processors are here, mostly infrastructure:
https://github.com/cython/cython/blob/master/Cython/Compiler/ParseTreeTransforms.py
It also contains the recursive unwinding of parallel assignments (a,b=c,d),
which leads to much better code at least at the C level. Not sure how
interesting something like this is for an interpreter. It might save some
tuples and/or stack operations along the way. Also, extended iterable
unpacking can be optimised this way.
Some builtins are generically optimised here (straight through the type
system instead of the AST):
https://github.com/cython/cython/blob/master/Cython/Compiler/Builtin.py
For refactoring the tree, we manually build tree nodes most of the time,
but for the more involved cases, we sometimes pass templates through the
parser.
The whole compiler pipeline, including all AST processing steps, is
constructed here:
https://github.com/cython/cython/blob/master/Cython/Compiler/Pipeline.py#L123
Stefan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] AST optimizer implemented in Python
"Martin v. Löwis", 11.08.2012 23:27: >>> +1 We should add some form of setastoptimizer API in 3.4. Please start a >>> PEP for this. It would be nice to include the ability to properly cache >>> the >>> ast optimizer output so that it does not have to run every time (in pyc >>> files or similar, etc) but can verify that it is the specific ast optimizer >>> that ran on a give existing cached copy. >> >> Once .pyc is created, I do not think that we keep the AST around any >> longer. Otherwise, I wouldn't have to write PyXfuscator. >> >> https://bitbucket.org/namn/pyxfuscator >> >> Or perhaps I am misunderstanding you. > > I think you misunderstood. What gps is concerned about (IIUC) that some > people add ast optimizers in some run of Python, but other AST optimizers > in a different run. Then, if you use a Python byte code > file, you should be able to find out what AST optimizers have been > run to create the pyc file, so you know whether you have to recompile > or not. > > Of course, if that is really a desirable feature, you may want multiple > pyc files, per combination of AST optimizers. The __pycache__ directory > would readily support that. > > This seems to get complicated quickly, so a PEP is indeed desirable. I think that gets overly complicated due to state explosion. Just recompile when things do not match, or simply ignore mismatches (possibly based on a command line switch). Compilers for statically compiled languages have been working nicely like that for ages. Most people won't run the interpreter with different optimiser settings, and if they do, they can just delete the __pycache__ before the next run. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
