Re: [Python-Dev] Memory management in the AST parser & compiler

[Nick Coghlan]

Greg Ewing wrote:
> Neal Norwitz wrote:
> 
>> I'm mostly convinced that using PyObjects would be a good thing. 
>> However, making the change isn't free as all the types need to be
>> created and this is likely quite a bit of code.
> 
> Since they're all so similar, perhaps they could be
> auto-generated by a fairly simple script?
> 
> (I'm being very careful not to suggest using Pyrex
> for this, as I can appreciate the desire not to make
> such a fundamental part of the core dependent on it!)

The ast C structs are already auto-generated by a Python script (asdl_c.py, to 
be precise). The trick is to make that script generate full PyObjects rather 
than the simple C structures that it generates now.

I believe Jeremy wrote that early in the life of the AST branch, so it's worth 
waiting for his advice on how to go about modifying it.

asdl_seq can disappear entirely: we can just use a PyList instead.

The second step is to then modify ast.c to use the new structures. A branch 
probably wouldn't help much with initial development (this is a "break the 
world, check in when stuff compiles again" kind of change, which is hard to 
split amongst multiple people), but I think it would be of benefit when 
reviewing the change before moving it back to the trunk.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Metaclass problem in the "with" statement semantics in PEP 343

[Nick Coghlan]

Edward Loper wrote:
> Otherwise, it seems like people might write code that relies on the 
> current behavior, which will then break if we eg turn __context__ into 
> a slot.  (It sounds like you want to reserve the right to change this.) 
>   Well, of course, people may rely on the current behavior anyway, but 
> at least they'll have been warned. :)

Yep - I thought "the instance dictionary has no effect" was an actual rule, 
but it turns out the rules are slightly looser than that (specifically, that 
fact that the effect of having a slot name in the instance dictionary is 
undefined).

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Michael Hudson]

Greg Ewing <[EMAIL PROTECTED]> writes:

> Guido van Rossum wrote:
>
>>>To me, that's an argument in favour of always generating
>>>a .pyc, even for scripts.
>> 
>> I'm not sure I follow the connection.
>
> You were saying that if the parser and compiler were
> slow, it would slow down single-file scripts that
> didn't have a .pyc (or at least that's what I thought
> you were saying). If a .pyc were always generated,
> this problem would not arise.

Well, the current stdlib compiler is unacceptably slow, no question.
I don't want "make install" to take as long as "regrtest -u all
test_compiler", or make test to take nearly that long in all cases.

Cheers,
mwh

-- 
58. Fools ignore complexity. Pragmatists suffer it. Some can avoid
it. Geniuses remove it.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Thomas Lee]

Nick Coghlan wrote:

>Greg Ewing wrote:
>  
>
>>Neal Norwitz wrote:
>>
>>
>>
>>>I'm mostly convinced that using PyObjects would be a good thing. 
>>>However, making the change isn't free as all the types need to be
>>>created and this is likely quite a bit of code.
>>>  
>>>
>>Since they're all so similar, perhaps they could be
>>auto-generated by a fairly simple script?
>>
>>(I'm being very careful not to suggest using Pyrex
>>for this, as I can appreciate the desire not to make
>>such a fundamental part of the core dependent on it!)
>>
>>
>
>The ast C structs are already auto-generated by a Python script (asdl_c.py, to 
>be precise). The trick is to make that script generate full PyObjects rather 
>than the simple C structures that it generates now.
>
>  
>
I was actually trying this approach last night. I'm back to it this 
evening, working with the ast-objects branch. I'll push a patch tonight 
with whatever I get done.

Quick semi-related question: where are the marshal_* functions called? 
They're all static in Python-ast.c and don't seem to be actually called 
anywhere. Can we ditch them?

>The second step is to then modify ast.c to use the new structures. A branch 
>probably wouldn't help much with initial development (this is a "break the 
>world, check in when stuff compiles again" kind of change, which is hard to 
>split amongst multiple people), but I think it would be of benefit when 
>reviewing the change before moving it back to the trunk.
>
>  
>
Based on my (limited) experience and your approach, compile.c may also 
need to be modified a little too (this should be pretty trivial).

Cheers,
Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[A.M. Kuchling]

On Wed, Nov 30, 2005 at 07:42:20PM +1000, Nick Coghlan wrote:
> The second step is to then modify ast.c to use the new structures. A branch 
> probably wouldn't help much with initial development (this is a "break the 
> world, check in when stuff compiles again" kind of change, which is hard to 
> split amongst multiple people), ...

There is a bug day scheduled for this Sunday, so maybe the AST
developers could meet to coordinate this change.

--amk

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed additional keyword argument in lo gging calls

[Thomas Heller]

Vinay Sajip <[EMAIL PROTECTED]> writes:

> The existing fields which could be overwritten are ones which have been 
> computed
> by the logging package itself:
>
> nameName of the logger
> levelno Numeric logging level for the message (DEBUG, INFO,
> WARNING, ERROR, CRITICAL)
[and so on].

Shouldn't this list be documented?  Or is it?

Thomas

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposed additional keyword argument in logging calls

[Jim Jewett]

> I couldn't think of a good reason why it should be possible to overwrite these
> values with values from a user-supplied dictionary, other than to spoof log
> entries in some way. The intention is to stop a user accidentally overwriting
> one of the above attributes.

This makes sense, but is it worth the time to check on each logging call?

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Neal Norwitz]

On 11/30/05, Thomas Lee <[EMAIL PROTECTED]> wrote:
>
> Quick semi-related question: where are the marshal_* functions called?
> They're all static in Python-ast.c and don't seem to be actually called
> anywhere. Can we ditch them?

I *think* they are not necessary.  My guess is that they were there
for marshaling the AST to disk, though I'm not sure why we would want
to do that.  It could have been there was the idea of how they would
be marshalled to PyObjects and exported.

Unless you hear otherwise from Jeremy, I would probably remove them.

I can check your patch into the branch so others can get an idea and
hopefully provide comments.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Neil Schemenauer]

Thomas Lee <[EMAIL PROTECTED]> wrote:
> Quick semi-related question: where are the marshal_* functions called? 
> They're all static in Python-ast.c and don't seem to be actually called 
> anywhere. Can we ditch them?

They are intended to be used to make the AST available to Python
code.  It would be nice if they could be retained but nothing will
break (AFAIK) if they are ditched.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

Hello,

I've not had much luck in searching for a discussion on this in the
Python-Dev archives, so bear with me.

I had an idea this morning for a simple extension to Python's iterator
protocol that would allow the user to force an iterator to raise
StopIteration on the next call to next().  My thought was to add a new
method to iterators called stop().

In my situation it would be useful as a control-flow mechanism, but I
imagine there are many other use cases for it:


generator = some_generator_function()

for x in generator:
... deeply ...
... nested ...
... control-flow ...

if satisfaction_condition:
# Terminates the for-loop, but
# finishes the current iteration
generator.stop()

... more stuff ...


I'm curious if anything like this has been proposed in the past.  If so,
could someone kindly point me to any relevant mailing list threads?

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Neal Norwitz]

On 11/30/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote:
> Thomas Lee <[EMAIL PROTECTED]> wrote:
> > Quick semi-related question: where are the marshal_* functions called?
> > They're all static in Python-ast.c and don't seem to be actually called
> > anywhere. Can we ditch them?
>
> They are intended to be used to make the AST available to Python
> code.  It would be nice if they could be retained but nothing will
> break (AFAIK) if they are ditched.

If everything is a PyObject, wouldn't they be redundant?

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Alex Martelli]

On 11/30/05, Matthew F. Barnes <[EMAIL PROTECTED]> wrote:
   ...
> I'm curious if anything like this has been proposed in the past.  If so,
> could someone kindly point me to any relevant mailing list threads?

PEP 342, already accepted and found at
http://python.org/peps/pep-0342.html , covers related functionality
(as well as many other points).


Akex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

On Wed, 2005-11-30 at 10:57 -0800, Alex Martelli wrote:
> PEP 342, already accepted and found at
> http://python.org/peps/pep-0342.html , covers related functionality
> (as well as many other points).

Thanks Alex, I'll take another look at that PEP.  The first time I tried
to read it my brain started to sizzle.

I happened to use a generator-iterator in my example, but my thought was
that the extension could be applied to iterators in general, including
sequence-iterators.

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Edward Loper]

> I had an idea this morning for a simple extension to Python's iterator
> protocol that would allow the user to force an iterator to raise
> StopIteration on the next call to next().  My thought was to add a new
> method to iterators called stop().

There's no need to change the iterator protocol for your example use 
case; you could just define a simple iterator-wrapper:

class InterruptableIterator:
 stopped = False
 def __init__(self, iter):
 self.iter = iter()
 def next(self):
 if stopped:
 raise StopIteration('iterator stopped.')
 return self.iter.next()
 def stop(self):
 self.stopped = True

And then just replace:
> generator = some_generator_function()
with:
 generator = InterruptableIterator(some_generator_function())

-Edward

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] something is wrong with test___all__

[Reinhold Birkenfeld]

Guido van Rossum wrote:
> Has this been handled yet? If not, perhaps showing the good and bad
> bytecode here would help trigger someone's brain into understanding
> the problem.

I've created a tracker item at www.python.org/sf/1370322.

Reinhold

-- 
Mail address is perfectly valid!

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

On Wed, 2005-11-30 at 14:36 -0500, Edward Loper wrote:
> There's no need to change the iterator protocol for your example use 
> case; you could just define a simple iterator-wrapper:

Good point.  Perhaps it would be a useful addition to the itertools
module then?

itertools.interruptable(iterable)

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Neil Schemenauer]

Neal Norwitz <[EMAIL PROTECTED]> wrote:
> If everything is a PyObject, wouldn't [the marshal functions] be
> redundant?

You could be right.  Spending time to kept them working is probably
wasted effort.

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Standalone email package in the sandbox

[Barry Warsaw]

Unless there are any objections, I'd like to create a space in the
sandbox for the standalone email package miscellany.  This currently
lives in the mimelib project's hidden CVS on SF, but that seems pretty
silly.  

Basically I'm just going to add the test script, setup.py, generated
html docs and a few additional unit tests, along with svn:external refs
to pull in Lib/email from the appropriate Python svn tree.  This way,
I'll be able to create standalone email packages from the sandbox (which
I need to do because I plan on fixing a few outstanding email bugs).

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python bug day this Sunday

[A.M. Kuchling]

Let's have a Python bug day this Sunday.  One goal might be
to assess bugs and patches, and make a list of ones we can work on at  
the Python core sprint at PyCon
.

Meeting on IRC: #python-dev on irc.freenode.net

Date: Sunday, December 4th

Time: roughly 9AM to 3PM Eastern (2PM to 8PM UTC).

People on the US West Coast may want to show up from 9AM to
3PM Pacific time (12PM to 6PM Eastern), because it'll be more
convenient.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] ast-objects branch created

[Martin v. Löwis]

I created

http://svn.python.org/projects/python/branches/ast-objects/

You can convert your repository to that branch with

svn switch svn+ssh://[EMAIL PROTECTED]/python/branches/ast-objects

in the toplevel directory. In particular, this features

http://svn.python.org/projects/python/branches/ast-objects/Parser/asdl_c.py
http://svn.python.org/projects/python/branches/ast-objects/Include/Python-ast.h
http://svn.python.org/projects/python/branches/ast-objects/Python/Python-ast.c

The status is currently this:
- asdl_c generates a type hierarchy: "Sum" productions have one type
   per constructor, inheriting from a type for the sum; plain products
   only have a type for the product.
- attributes are in the base type, accessible through o->_base.attr;
   projections of the product types are accessible directly through
   member names.
- all projections must be non-NULL. Sequences are represented through
   potentially empty lists; optional types are potentially represented
   through Py_None. bool is gone; use Py_True/Py_False. The only
   primitive type remaining is int (which only occurs in lineno)
- the types currently have only a constructor, a dealloc function,
   and an _Check macro.
- Naming is this: for cross-object-file visible symbols (functions
   and global variables), a Py_ prefix is used. Otherwise, I use
   the type name or constructor name directly. There is a #define
   for the Py__New function, so you can also write (params).
   Parameter order for the types is: projections first, then
   attributes.
- For compatibility with the current code, the Sum base types
   also have the _kind enumeration (although that appears not
   to get initialized right now).

For asdl_c, I see the following things as TODOs:
- add support for traversing the types from C, through tp_members
   (read-only). Optionally add support for pickling.
- add support for garbage collection. I don't expect this to be
   necessary right now, but will be if the API is exposed, and it
   doesn't cost much.

The bigger chunk of necessary changes is in using these, starting
with ast.c.

Feel free to commit any changes to that branch that you consider
helpful. To avoid duplicated work, posting a note here might
also help.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Martin v. Löwis]

Nick Coghlan wrote:
 > The ast C structs are already auto-generated by a Python script 
(asdl_c.py, to
 > be precise). The trick is to make that script generate full PyObjects 
rather
 > than the simple C structures that it generates now.

See the ast-object branch.

 > The second step is to then modify ast.c to use the new structures. A 
branch
 > probably wouldn't help much with initial development (this is a 
"break the
 > world, check in when stuff compiles again" kind of change, which is 
hard to
 > split amongst multiple people), but I think it would be of benefit when
 > reviewing the change before moving it back to the trunk.

Well, there would be a clear two-split right now: one could change
ast.c, and the other compile.c.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Jeremy Hylton]

On 11/30/05, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> On 11/30/05, Thomas Lee <[EMAIL PROTECTED]> wrote:
> >
> > Quick semi-related question: where are the marshal_* functions called?
> > They're all static in Python-ast.c and don't seem to be actually called
> > anywhere. Can we ditch them?
>
> I *think* they are not necessary.  My guess is that they were there
> for marshaling the AST to disk, though I'm not sure why we would want
> to do that.  It could have been there was the idea of how they would
> be marshalled to PyObjects and exported.
>
> Unless you hear otherwise from Jeremy, I would probably remove them.
>
> I can check your patch into the branch so others can get an idea and
> hopefully provide comments.

The intent was to share the AST objects between C and Python by coping
them.  I still think passing copies is better than sharing live
objects between Python and C, although the specific mechanism may be
different if the C objects are PyObjects.

Jeremy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Greg Ewing]

Jeremy Hylton wrote:

> I still think passing copies is better than sharing live
> objects between Python and C,

Even if the objects are immutable?

-- 
Greg Ewing, Computer Science Dept, +--+
University of Canterbury,  | A citizen of NewZealandCorp, a   |
Christchurch, New Zealand  | wholly-owned subsidiary of USA Inc.  |
[EMAIL PROTECTED]  +--+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Memory management in the AST parser & compiler

[Jeremy Hylton]

Sure.  If they're immutable sharing is fine, but you end up making a
copy anyway if you want to make changes, right?

Jeremy

On 11/30/05, Greg Ewing <[EMAIL PROTECTED]> wrote:
> Jeremy Hylton wrote:
>
> > I still think passing copies is better than sharing live
> > objects between Python and C,
>
> Even if the objects are immutable?
>
> --
> Greg Ewing, Computer Science Dept, +--+
> University of Canterbury,  | A citizen of NewZealandCorp, a   |
> Christchurch, New Zealand  | wholly-owned subsidiary of USA Inc.  |
> [EMAIL PROTECTED]+--+
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast-objects branch created

[Neal Norwitz]

On 11/30/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
>
> The bigger chunk of necessary changes is in using these, starting
> with ast.c.

I got a few more files to compile.  The following files (all under
Python/) need some loving care and are looking for a kind soul to
adopt them:

  ast.c, compile.c, future.c, symtable.c

Of these, future.c is by far the easiest to get compiling.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast-objects branch created

[Brett Cannon]

On 11/30/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> I created
>
> http://svn.python.org/projects/python/branches/ast-objects/
>
> You can convert your repository to that branch with
>
> svn switch svn+ssh://[EMAIL PROTECTED]/python/branches/ast-objects
>

If you would rather do a separate checkout, do

 svn checkout svn+ssh://[EMAIL PROTECTED]/python/branches/ast-objects

If you want a read-only checkout, see the newly updated entry on
checking out projects in the dev FAQ at
http://www.python.org/dev/devfaq.html#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write
.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com