Re: [Python-Dev] PEP 343 and __context__()

[Nick Coghlan]

Jason Orendorff wrote:
> I just noticed that my name is in PEP 343 attached to the idea of the
> __context__() method, and I'm slightly queasy over it.
> 
> The rationale was to help e.g. decimal.DecimalContext support 'with'. 
> Maybe that's a bad idea.
> 
> DecimalContext has a few problems.  In code where it matters, every
> function you write has to worry about it. (That is, you can't just
> write __decimal_context__ = ... at the top of the file and be done
> with it, the way you can with, say, __metaclass__.)

No, you write "decimal.setcontext(...)" instead. You then only need to worry 
if you have multiple threads, and you can even fix that by modifying 
decimal.DefaultContext while the program is still single threaded. The latter 
approach actually works even for a single-threaded program, but should only be 
done in an application script, never in an imported module.

>  And
> DecimalContext doesn't fit in with generators.

It does fit actually - you simply have to remember to restore the original 
context around any invocations of yield.

> sys.stdout has similar problems.
> 
> It feels like PEP 343 will make people want to follow this model. 
> That is, we'll see more global behavior-controlling variables in the
> future.  There are grizzlier fates; I just wondered if anyone had
> thought of this.

Yeah, it came up in response to PJE's suggestion of task-local variables for 
generators. The basic concept we came up with is that if you're writing a 
generator that uses a specific context, remember to save the original and 
restore it around any calls to yield (at least, I came up with the idea and 
no-one objected to the approach [1]).

In the case of decimal.Context:

  def iter_sin(iterable):
 orig_ctx = decimal.getcontext()
 with orig_ctx as ctx:
 ctx.prec += 10
 for r in iterable:
 y = sin(r) # Very high precision during calculation
 with orig_ctx:
 yield +y # Yielded results have normal precision
 # We get "ctx" back here
 # We get "orig_ctx" back here

Similarly for sys.stdout (only not thread-safe due to the global state):

  def log_yielded_items(iterable, logfile):
 orig_stdout = sys.stdout
 with redirected_stdout(logfile):
 for r in iterable:
 print "Yielding:", r
 with redirected_stdout(orig_stdout):
 yield r # stdout goes back to normal here
 # stdout is going to the logfile again here
 # And stdout is back to normal at the end


The key properties of a "well-behaved" context are:
   1. Use thread-local state for contexts in order to be multithreading safe
 decimal.Context obeys this, sys.stdout doesn't

   2. Provide a way to retrieve the current state for explicit restoration
 decimal.getcontext() and sys.stdout allow this as shown above

This kind of thing is always going to be a pain, but PEP 343 makes it 
significantly less so.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2005-October/057493.html

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pystate.c changes for Python 2.4.2

[Michael Hudson]

Tim Peters <[EMAIL PROTECTED]> writes:

> [Gabriel Becedillas]
>> Can anybody tell me if the patch I suggested is ok ?
>> That will be to add the following code at the end of PyThreadState_Delete:
>>
>> if (autoTLSkey && PyThread_get_key_value(autoTLSkey) == tstate)
>>  PyThread_delete_key_value(autoTLSkey);
>
> It needs a little work, but I think it should be fine, and people tend
> to listen to me on issues like this ;-)

Oh good, I've been meaning to scrape together the brain cells to think
about this but PyPy is pulping them as fast as I find them...

> The reason it can't work as-is is shallow:  autoTLSkey doesn't exist
> unless Python is compiled with WITH_THREAD defined.  So you need to
> wrap that inside an "#ifdef WITH_THREAD" block; or add it similarly to
> tstate_delete_common(), and remove it from
> PyThreadState_DeleteCurrent().
>
> The primary reason people find this so confusing is because it is :-).
>  In the dim mists of history, neither multiple interpreters nor
> PyThreadState_DeleteCurrent() existed.  While multiple interpreters do
> exist now, they're poorly understood, and the PEP that introduced the
> GIL state API explicitly disavowed any responsibility for the new API
> working at all with multiple interpreters:
>
> http://www.python.org/peps/pep-0311.html
>
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState".  There is no attempt to cater for
> extensions that require multiple interpreter states.  At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
>
> In a parallel development, thread races during Python shutdown were
> discovered that could lead to segfaults, and
> PyThreadState_DeleteCurrent() was introduced to repair those.  As a
> result, it so happens that core Python never uses the original
> PyThreadState_Delete() anymore, except when Py_NewInterpreter() has to
> throw away the brand new thread state it created because it turns out
> it can't create a new interpreter.

Um, PyThreadState_Delete() is called from zapthreads() is called from
PyInterpreterState_Delete() is called from Py_Finalize()... so I don't
entirely believe you here :)

> Since core Python never calls Py_NewInterpreter() or
> PyThreadState_Delete(), you're in an intersection of areas no current
> Python developer even thinks about, let alone tests.  But by the same
> token, _because_ it's unused, there's nothing you can do to
> PyThreadState_Delete() that can hurt core Python either.  That's why
> people should be more encouraging here ;-)  It certainly makes sense
> to me that if a thread state is going away, any record of it in the
> auto-GIL-state machinery must also go away.

I guess if the patch fixes the original problem, it should go in -- my
uneasiness stems not from worrying that it might do harm, but rather
from the fact that I think it's incomplete.  Maybe I just haven't
thought it through enough.

Also, every time I look at pystate.c, I think of things that could be
done differently or better -- for example, I think it would be sane
and possibly even sensible to only call PyThread_set_key_value() in
_PyGILState_NoteThreadState() if "tstate->interp ==
autoInterpreterState".

>> Thank you very much.
>
> Thank you!  If you put a patch on SourceForge, I'll probably be happy
> to check it in.

That would get me off the "svn blame" hook :)

Cheers,
mwh

-- 
  My first thought was someone should offer Mr. Bush elocution
  lessons.  But he'd probably say he knew how to work them chairs
  already.-- Internet Oracularity #1294-01
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] site triggering a bug in urllib2

[Thomas Mangin]

Hello,

I am contacting the list in the hope that someone will be able to understand 
what I am seeing.


I have hit a bug with python 2.4.2 (on Mandriva 2006) using urllib2.
The code which trigger the bug is as follow..

import urllib2
req = urllib2.Request("http://66.117.37.13/";)

# makes no difference ..
req.add_header('Connection', 'close')

handle = urllib2.urlopen(req)
data = handle.read()
print data

using a timeout on the socket does not work neither.

The page is well sent and received (verified with tcpdump) but the read call is 
never returning. strace shows lots of activity on the socket: select constantly 
returning a file descriptor with nothing to read on.


I would like to avoid to start to use timers to kill the connection manually, 
any advise on how to deal with this would be welcome.


Thank you for your time and help.

Thomas
--
Exa Networks Limited - UK - AS30740 - www.exa-networks.co.uk
nic-handle : MANG-RIPE   website  : thomas.mangin.me.uk
GPG key ID : 0xFB8B81A1  PGP key  : /pgp.html
Inoc-DBA # : 30740*TOM   Office # : +44 (0) 845 145 1234


signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] timeit module

[Connelly Barnes]

Connelly Barnes wrote:
>> Hi,
>> 
>> Perhaps I am the only one bothered by the timeit
>> module, but it seems poorly designed to me.
>> 
>> First of all, it should use a geometric series with a
>> timeout value to detect how many iterations it should
>> perform.  Currently, the user is required to manually
>> specify the number of iterations (the default is 1
>
>The provision of a default value generally seems to chop the legs from

>your argument that the number of iterations is required.

Suppose we wish to time two functions: f1 and f2.  Function f1 takes
1e-5
seconds to run, and function f2 takes 0.1 seconds to run.  If these
functions
are defined in interactive mode, we can time them with timeit via:

>>> timeit.Timer('f1()', 'from __main__ import
f1').timeit(100)/100
>>> timeit.Timer('f2()', 'from __main__ import f2').timeit(10)/10

The first line gives the approximate time to run f1 and the second line
gives
the approximate time to run f2.  The number parameter to timeit() must
be
supplied for the second line, since the default 1 million iterations
would
take too long to be practical.  The number parameter to timeit() is
also
supplied for the first line, since it makes the code more robust in
case
the default argument value in timeit() is changed.

My first observation is that this is a lot of contortion to get the
time of
functions f1() and f2().  Something simpler would be nice.  For
example,
using Recipe 440657 this can be simplified to:

>>> pytime.pytime(f1, ())
>>> pytime.pytime(f2, ())

Here the timing routine calculates the number of iterations to make,
and
the timing routine divides by the number of iterations of the supplied
callable.

This is also convenient if you move from a computer to one that is
faster or slower.  If you move to a computer that is 10 times slower,
then the timeit.Timer() code above will take 10 times as long to run.
The usual solution is for the programmer to adjust the number of
iterations as he changes computers (so he doesn't wait around all day
for his benchmarks to run).  Likewise, if one optimizes f2() so that
it runs 100 times faster, one would again have to adjust the number
of iterations in order to gain an accurate timing result (if the total
time spent in the timing routine is less than 1 msec, then the
resulting time for f2() is off by more than 10% on my Windows machine,
which is undesirable, so the number of iterations cannot be made too
small).

Summary:
Module timeit is complicated to use on an existing function, it does
not time the number of seconds to execute 1 iteration of the function
(this is the value one usually wants in benchmarking, or its
reciprocal), and it requires that the end user tweak the number of
iterations as he or she changes between computers with
different CPU speed, or as he or she optimizes code.  It is easy to
make a cleaner module that fixes these problems.  It might be good
to incorporate such a module into Python, due to its general
usefulness.

>> million).  If the user optimizes his or her code, then
>> the number of iterations must be changed.  If the user
>> moves to a slower or faster computer, then the number
>> of iterations must be changed again.  This is
>> annoying.
>> 
>What? The purpose of timeit is to give an approximation to the length
of 
>time to run a specific piece ofcode.
>
>Why must the number of iterations be changed when moving to a slower
or 
>faster computer?
>
>> Secondly, there should be a way to time a callable
>> directly.  That is, without finding the string name of
>> the callable and using a string "import X" statement. 
>> These contortions violate rules #1 and #3 of the Zen
>> of Python.
>> 
>Presumably for benchmarking purposes the function call overhead would
be 
>present for all compaered techniques. Do you mean rules #0 and #2?

I mean "Beautiful is better than ugly" and "Simple is better than
complex."


> > [...]
>regards
>  Steve
>-- 
>Steve Holden   +44 150 684 7255  +1 800 494 3119
>Holden Web LLC www.holdenweb.com
>PyCon TX 2006  www.python.org/pycon/


__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ConfigParser to save with order

[Tony Meyer]

[Tony Meyer]
> Allowing 'surgical' editing of configuration files, as has been
> proposed many times both here and c.l.p would not require
> ConfigParser to be entirely rewritten (just more extensive
> modification of the write() method).

After writing the summary of this thread, I figured I might as well  
prove this by submitting a patch, so did so:

[ 1410680 ] Add 'surgical editing' to ConfigParser


So there are now at least three open ConfigParser patches, all  
addressing similar things, although not in the same way, and all  
three could actually be applied.

My patch (which includes new unittests and a marked-up doc patch)  
runs through a file and updates it to match the in-memory  
configuration object.  Comments, blank lines, and order are  
preserved.  By default (this can be turned off with a keyword arg)  
new sections/options are added to the file as well (new options in  
existing sections are added at the end of the last instance of that  
section, new sections are added at the end of the file).  These new  
options are added in alphabetical (well, list.sort()) order, so  
ordering is consistent, and if an empty file is 'updated' one ends up  
with the output of write() but sorted.

This is essentially a tidied-up version of the SpamBayes code I wrote  
a few years back.

If anyone is still interested in adding this, feel free to take a  
look at the tracker :)

=Tony.Meyer
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] str with base

[Stephen J. Turnbull]

> "BAW" == Barry Warsaw <[EMAIL PROTECTED]> writes:

BAW> Unix weenies shouldn't be totally forgotten in P3K.

Great idea!  Put all this stuff in a "weenie" module.  You can have
weenie.unix and weenie.vms and weenie.unicode, besides the weenie.math
that got all this started.

-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of TsukubaTennodai 1-1-1 Tsukuba 305-8573 JAPAN
   Ask not how you can "do" free software business;
  ask what your business can "do for" free software.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ConfigParser to save with order

[Paramjit Oberoi]

> I think it's moot unless you also preserve comments. Ideally would be
> something that prserved everything (ordering, blank lines, comments
> etc.) from how it was read in. Modifying a value should keep its
> position. Adding a value should add it to the end of the section it's
> in (unless there are cases where ordering is important to the
> semantics -- are there?).

I wrote some code to do just that sometime back:

http://www.cs.wisc.edu/~param/software/cfgparse/

It would probably need some cleanup and restructuring to be used, but
it may be a good starting point if configparser is to be rewritten.

-param
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] timeit module

[Nick Coghlan]

Steve Holden wrote:
> Connelly Barnes wrote:
>> Hi,
>>
>> Perhaps I am the only one bothered by the timeit
>> module, but it seems poorly designed to me.
>>
>> First of all, it should use a geometric series with a
>> timeout value to detect how many iterations it should
>> perform.  Currently, the user is required to manually
>> specify the number of iterations (the default is 1
> 
> The provision of a default value generally seems to chop the legs from 
> your argument that the number of iterations is required.

 From the command line, timeit autocalibrates to use the smallest power of ten 
that results in taking at least 0.2 seconds to execute all iterations.

IMO, making this the default behaviour of the timer objects themselves would 
be *much* more useful behaviour than the current default to 10e6 iterations. 
I've been caught a few times hammering on Ctrl-C when I realised that I'd just 
told the interpreter to go do a million iterations of something that took a 
few seconds for each attempt.

>> million).  If the user optimizes his or her code, then
>> the number of iterations must be changed.  If the user
>> moves to a slower or faster computer, then the number
>> of iterations must be changed again.  This is
>> annoying.
>>
> What? The purpose of timeit is to give an approximation to the length of 
> time to run a specific piece ofcode.
> 
> Why must the number of iterations be changed when moving to a slower or 
> faster computer?

Because the default of 10e6 may be entirely inappropriate depending on how 
long an individual iteration is. If a single iteration takes a microsecond, 
great, but if it takes a second, better send out for pizza (or maybe go for a 
trip around the world in the 4 months that sucker is going to take to execute!)

If Timer was changed as follows, it would "just work", even when each 
iteration took a few seconds (6 seconds per iteration would give 1 minute for 
each execution of the timeit method):

class Timer:
 def __init__(self, stmt="pass", setup="pass", timer=default_timer):
 """Constructor.  See class doc string."""
 # ... existing constructor
 self.default_number = None

 def timeit(self, number=None): # Note changed default
 if number is None:
 if self.default_number is None:
 self.default_number = self.calibrate()
 number = self.default_number
 # ... existing timeit method

 def calibrate(self): # New method, taken from script code at end of module
 # find smallest power of 10 >= 10 so that 0.2 seconds <= total time
 # capped at 10 billion (10e10) iterations, as reaching that limit
 # implies fewer than 20 picoseconds (2e-11) per iteration (i.e. Fast!)
 for i in range(1, 10):
 number = 10**i
 x = self.timeit(number)
 if x >= 0.2:
 break
 self.default_number = number


>> Secondly, there should be a way to time a callable
>> directly.  That is, without finding the string name of
>> the callable and using a string "import X" statement. 
>> These contortions violate rules #1 and #3 of the Zen
>> of Python.
>>
> Presumably for benchmarking purposes the function call overhead would be 
> present for all compaered techniques. Do you mean rules #0 and #2?

Timing an existing function really is a pain - timeit expects source code it 
can plug into a code template, so timing an existing function from an 
interactive session isn't as easy as it could be (you have to figure out a way 
to give strings to Timer that it can plug into its string template, rather 
than just giving it the callable directly).

A subclass would deal with that quite handily:

class CallTimer(Timer):
 # Use lambda or functional.partial to pass arguments to the callable
 def __init__(self, callable, timer=default_timer):
 def inner(_it, _timer):
 _t0 = _timer()
 for _i in _it:
 callable()
 _t1 = _timer()
 return _t1 - _t0
 self.timer = timer
 self.inner = inner
 self.src = None

 # Using a real function, so leave linecache alone when printing exceptions
 def print_exc(self, file=None):
 import traceback
 traceback.print_exc(file=file)

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pystate.c changes for Python 2.4.2

[Tim Peters]

[Tim]
>> ...
>> As a result, it so happens that core Python never uses the original
>> PyThreadState_Delete() anymore, except when Py_NewInterpreter() has
>> to throw away the brand new thread state it created because it turns out
>> it can't create a new interpreter.

[Michael]
> Um, PyThreadState_Delete() is called from zapthreads() is called from
> PyInterpreterState_Delete() is called from Py_Finalize()... so I don't
> entirely believe you here :)

That's only because I was wrong ;-)  Thanks!  That use was in fact
half the cause of Python's segfaulting shutdown races, so it was
exceptionally stupid of me to miss that one.

>> Since core Python never calls Py_NewInterpreter() or
>> PyThreadState_Delete(), you're in an intersection of areas no current
>> Python developer even thinks about, let alone tests.  But by the same
>> token, _because_ it's unused, there's nothing you can do to
>> PyThreadState_Delete() that can hurt core Python either.  That's why
>> people should be more encouraging here ;-)  It certainly makes sense
>> to me that if a thread state is going away, any record of it in the
>> auto-GIL-state machinery must also go away.

> I guess if the patch fixes the original problem, it should go in -- my
> uneasiness stems not from worrying that it might do harm, but rather
> from the fact that I think it's incomplete.  Maybe I just haven't
> thought it through enough.

Oh, I don't know whether it's "complete".  I don't really understand
multiple interpreters, have no idea why the OP is calling
PyThreadState_Delete(), and fully expect that nobody is going to add
tests to Python to check any of this.  But Gabriel showed real skill
at tracking down problems in this area, so if he creates new problems
for himself by doing this (who else plays in this intersection? 
nobody I know of), I'm sure he'll figure them out ;-)

For example, it seems possible that he'll hit the same kinds of
segfaulting shutdown races the Python core used to suffer.  If a
thread T releases the GIL, and goes on to delete its own thread state
via PyThreadState_Delete(T's_thread_state), that _is_ a race with
Python shutdown's zapthreads trying to delete the same thing.  That's
why PyThreadState_DeleteCurrent(void) was introduced, and any call to
PyThreadState_Delete() is suspect on this count.

> Also, every time I look at pystate.c, I think of things that could be
> done differently or better -- for example, I think it would be sane
> and possibly even sensible to only call PyThread_set_key_value() in
> _PyGILState_NoteThreadState() if "tstate->interp ==
> autoInterpreterState".

I joined Mark Hammond in ignoring the possibility of multiple
interpreters when this stuff went in.  The best thing to do with
untested gimmicks without an audience willing to work on them is to
remove them.  If you want to "rehabilitate" multiple interpreters
here, the best first step would be to write an addendum to PEP 311
documenting an intended design.  The GIL state API is documented well
enough on its own, but the _intended_ semantics when mixing threads
with multiple interpreters is clear as mud.  PEP 311 asked "but who
cares?", and nobody answered.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 343 and __context__()

[Phillip J. Eby]

At 07:21 PM 01/20/2006 +1000, Nick Coghlan wrote:
>Yeah, it came up in response to PJE's suggestion of task-local variables for
>generators. The basic concept we came up with is that if you're writing a
>generator that uses a specific context, remember to save the original and
>restore it around any calls to yield (at least, I came up with the idea and
>no-one objected to the approach [1]).

That's only because Guido shut down the thread.  :)

For the use cases I had in mind, your approach simply doesn't scale, 
because it requires every yield-point to have global knowledge of all 
"changes to global state" that may have occurred somewhere above it in the 
coroutine call stack.  The only reason I didn't point this out is that 
Guido had asked for a postponement of the discussion.

Since then, I've begun implementing a library that accomodates such use 
cases, as well as a wide variety of other context-management use 
cases.  Source code is at http://svn.eby-sarna.com/Contextual/ and the docs 
are at http://svn.eby-sarna.com/Contextual/context.txt?view=markup

The library uses the current draft of PEP 343 as the basis for 
implementation, and includes a few helper functions that can be used to 
emulate the "with:" statement in Python 2.4 so that it can be used and 
tested.  So far, my impression is that the current 343 spec is at least 
adequate, and maybe even elegant.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] yield back-and-forth?

[Guido van Rossum]

The discussion about PEP 343 reminds me of the following. Bram Cohen
pointed out in private email that, before PEP 342, there wasn't a big
need for a shortcut to pass control to a "sub-generator" because the
following for-loop works well enough:

 def main_generator():
  ...
  for value in sub_generator():
  yield value

but now that yield can return a value, that value might have to be
passed into sub_generator(), not to mention of exceptions. I'm sure
there's a way to write that (although I haven't found the time to
figure it out) but I expect it to be cumbersome and subtle. I don't
think a helper function can be created to solve this problem, because
the yield syntax is essential.

Bram suggested the following syntax:

  def main_generator():
  ...
  yieldthrough sub_generator()

I'm not keen on that particular keyword, but I do believe a syntactic
solution is needed, if the problem is important enough to be solved.

Thoughts?

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] site triggering a bug in urllib2

[Aahz]

On Tue, Jan 17, 2006, Thomas Mangin wrote:
>
> I am contacting the list in the hope that someone will be able to 
> understand what I am seeing.

You'll probably get more help by subscribing and posting to
comp.lang.python.
-- 
Aahz ([EMAIL PROTECTED])   <*> http://www.pythoncraft.com/

"19. A language that doesn't affect the way you think about programming,
is not worth knowing."  --Alan Perlis
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Christian Tanzer]

Guido van Rossum <[EMAIL PROTECTED]> wrote:

> The discussion about PEP 343 reminds me of the following. Bram Cohen
> pointed out in private email that, before PEP 342, there wasn't a big
> need for a shortcut to pass control to a "sub-generator" because the
> following for-loop works well enough:
>
>  def main_generator():
>   ...
>   for value in sub_generator():
>   yield value

For small values of `well enough`. I sometimes override a generator in
a subclass and it sucks to add the loop just because the derived
generator wants to add a value at the beginning or end.

> but now that yield can return a value, that value might have to be
> passed into sub_generator(), not to mention of exceptions. I'm sure
> there's a way to write that (although I haven't found the time to
> figure it out) but I expect it to be cumbersome and subtle. I don't
> think a helper function can be created to solve this problem, because
> the yield syntax is essential.
>
> Bram suggested the following syntax:
>
>   def main_generator():
>   ...
>   yieldthrough sub_generator()
>
> I'm not keen on that particular keyword, but I do believe a syntactic
> solution is needed, if the problem is important enough to be solved.

How about:

  def main_generator():
  ...
  yield * sub_generator()

Ducking-ly yrs,

-- 
Christian Tanzerhttp://www.c-tanzer.at/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Phillip J. Eby]

At 10:17 AM 01/20/2006 -0800, Guido van Rossum wrote:
>The discussion about PEP 343 reminds me of the following. Bram Cohen
>pointed out in private email that, before PEP 342, there wasn't a big
>need for a shortcut to pass control to a "sub-generator" because the
>following for-loop works well enough:
>
>  def main_generator():
>   ...
>   for value in sub_generator():
>   yield value
>
>but now that yield can return a value, that value might have to be
>passed into sub_generator(), not to mention of exceptions. I'm sure
>there's a way to write that (although I haven't found the time to
>figure it out) but I expect it to be cumbersome and subtle. I don't
>think a helper function can be created to solve this problem, because
>the yield syntax is essential.
>
>Bram suggested the following syntax:
>
>   def main_generator():
>   ...
>   yieldthrough sub_generator()
>
>I'm not keen on that particular keyword, but I do believe a syntactic
>solution is needed, if the problem is important enough to be solved.

What's the use case for this?  In the coroutine use case, the PEP 342 
sample trampoline takes care of this.  If you yield a sub-generator (i.e. 
'yield sub_generator()'), the trampoline effectively replaces the parent 
with the child until the child is complete.

So, that leaves only non-coroutine use cases, and I'm having a hard time 
thinking of any where there would be bidirectional communication.


>Thoughts?

If we have to have a syntax, "yield from sub_generator()" seems clearer 
than "yieldthrough", and doesn't require a new keyword.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] site triggering a bug in urllib2

[Bill Janssen]

Or the Web-SIG mailing list.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Guido van Rossum]

On 1/20/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> At 10:17 AM 01/20/2006 -0800, Guido van Rossum wrote:
> >The discussion about PEP 343 reminds me of the following. Bram Cohen
> >pointed out in private email that, before PEP 342, there wasn't a big
> >need for a shortcut to pass control to a "sub-generator" because the
> >following for-loop works well enough:
> >
> >  def main_generator():
> >   ...
> >   for value in sub_generator():
> >   yield value
> >
> >but now that yield can return a value, that value might have to be
> >passed into sub_generator(), not to mention of exceptions. I'm sure
> >there's a way to write that (although I haven't found the time to
> >figure it out) but I expect it to be cumbersome and subtle. I don't
> >think a helper function can be created to solve this problem, because
> >the yield syntax is essential.
> >
> >Bram suggested the following syntax:
> >
> >   def main_generator():
> >   ...
> >   yieldthrough sub_generator()
> >
> >I'm not keen on that particular keyword, but I do believe a syntactic
> >solution is needed, if the problem is important enough to be solved.
>
> What's the use case for this?  In the coroutine use case, the PEP 342
> sample trampoline takes care of this.  If you yield a sub-generator (i.e.
> 'yield sub_generator()'), the trampoline effectively replaces the parent
> with the child until the child is complete.

That's a rather specialized, subtle and elaborate framework though,
and at this time I believe it isn't planned to be part of Python 2.5
(right?).

> So, that leaves only non-coroutine use cases, and I'm having a hard time
> thinking of any where there would be bidirectional communication.

Any other use of generators where the return value of yield is used;
as soon as you do this you may later want to refactor the code to to
include sub-iterators. (There *are*other uses besides the trampoline,
right? :-)

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Edward C. Jones]

Guido van Rossum <[EMAIL PROTECTED]> wrote:

The discussion about PEP 343 reminds me of the following. Bram Cohen
pointed out in private email that, before PEP 342, there wasn't a big
need for a shortcut to pass control to a "sub-generator" because the
following for-loop works well enough:

 def main_generator():
  ...
  for value in sub_generator():
  yield value

This is an important programming trick. It allows recursive use of 
generators for walking trees, etc. See the source code for os.walk for 
an example.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] site triggering a bug in urllib2

[John J Lee]

On Tue, 17 Jan 2006, Thomas Mangin wrote:
[...]
> I have hit a bug with python 2.4.2 (on Mandriva 2006) using urllib2.
> The code which trigger the bug is as follow..
>
> import urllib2
> req = urllib2.Request("http://66.117.37.13/";)
>
> # makes no difference ..
> req.add_header('Connection', 'close')
>
> handle = urllib2.urlopen(req)
> data = handle.read()
> print data
>
> using a timeout on the socket does not work neither.

This is a real bug, I think.  I filed a report on the SF bug tracker:

http://python.org/sf/1411097


The problem seems to be the (ab)use of socket._fileobject in urllib2 (I 
believe this was introduced when urllib2 switched to using 
httplib.HTTPConnection).  The purpose of the hack (as commented in 
AbstractHTTPHandler.do_open()) is to provide .readline() and .readlines() 
methods on the response object returned by urllib2.urlopen().

Workaround if you're not using .readline() or .readlines() (against 2.4.2, 
but should apply against current SVN):

--- urllib2.py.orig Fri Jan 20 20:10:56 2006
+++ urllib2.py  Fri Jan 20 20:12:07 2006
@@ -1006,8 +1006,7 @@
  # XXX It might be better to extract the read buffering code
  # out of socket._fileobject() and into a base class.

-r.recv = r.read
-fp = socket._fileobject(r)
+fp = r.fp

  resp = addinfourl(fp, r.msg, req.get_full_url())
  resp.code = r.status


Not sure yet what the actual problem/cure is...


John
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 343 and __context__()

[Jason Orendorff]

On 1/20/06, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Jason Orendorff wrote:
> > DecimalContext has a few problems.  In code where it matters, every
> > function you write has to worry about it. (That is, you can't just
> > write __decimal_context__ = ... at the top of the file and be done
> > with it, the way you can with, say, __metaclass__.)
>
> No, you write "decimal.setcontext(...)" instead.

You seem to be implying these are roughly equal in convenience; I
disagree.  Suppose I have banking.py, in which it's important to use a
particular precision and rounding.  Now I have to put context-munging
code in every single function that banking.py exposes.  And around
every 'yield'.  Even with 'with', that's a lot of extra lines of code.

I'd much prefer to put a one-liner at the top of the file, if it were
possible (...but I don't see how, yet).

Again, none of this is likely to matter--unless you're interleaving
banking and heavy scientific calculations, which I try to avoid.  So,
not a big deal.  Thanks for the response.

> >  And
> > DecimalContext doesn't fit in with generators.
>
> It does fit actually - you simply have to remember to restore the original
> context around any invocations of yield.

Feh!  "Fit" is to "can be made to work with a bit of effort, just
don't forget to follow the rules" as Python is to C++.

-j
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r42116 - python/branches/release24-maint/Lib/unittest.py

[Thomas Wouters]

On Fri, Jan 20, 2006 at 06:55:03PM +0100, georg.brandl wrote:
> Author: georg.brandl
> Date: Fri Jan 20 18:55:02 2006
> New Revision: 42116
> 
> Modified:
>python/branches/release24-maint/Lib/unittest.py
> Log:
> Patch #1388073: Make unittest.TestCase easier to subclass

I don't believe this belongs in 2.4, since it can, actually, break code.
Code that depends on the current situation, _TestCase__attributename.
Fragile code, to be sure, but still. If there were a compelling reason to
backport, I guess it could be hacked to work right-ish, but subclassing
TestCase in this way, while convenient, isn't important enough to warrant
this (IMHO).

-- 
Thomas Wouters <[EMAIL PROTECTED]>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r42116 - python/branches/release24-maint/Lib/unittest.py

[Guido van Rossum]

Right. This *definitely* is a feature, not a bug.

On 1/20/06, Thomas Wouters <[EMAIL PROTECTED]> wrote:
> On Fri, Jan 20, 2006 at 06:55:03PM +0100, georg.brandl wrote:
> > Author: georg.brandl
> > Date: Fri Jan 20 18:55:02 2006
> > New Revision: 42116
> >
> > Modified:
> >python/branches/release24-maint/Lib/unittest.py
> > Log:
> > Patch #1388073: Make unittest.TestCase easier to subclass
>
> I don't believe this belongs in 2.4, since it can, actually, break code.
> Code that depends on the current situation, _TestCase__attributename.
> Fragile code, to be sure, but still. If there were a compelling reason to
> backport, I guess it could be hacked to work right-ish, but subclassing
> TestCase in this way, while convenient, isn't important enough to warrant
> this (IMHO).
>
> --
> Thomas Wouters <[EMAIL PROTECTED]>
>
> Hi! I'm a .signature virus! copy me into your .signature file to help me 
> spread!
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r42116 - python/branches/release24-maint/Lib/unittest.py

[Barry Warsaw]

On Fri, 2006-01-20 at 21:43 +0100, Thomas Wouters wrote:

> I don't believe this belongs in 2.4, since it can, actually, break code.
> Code that depends on the current situation, _TestCase__attributename.
> Fragile code, to be sure, but still. If there were a compelling reason to
> backport, I guess it could be hacked to work right-ish, but subclassing
> TestCase in this way, while convenient, isn't important enough to warrant
> this (IMHO).

Exactly right.
-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r42116 - python/branches/release24-maint/Lib/unittest.py

[Georg Brandl]

Barry Warsaw wrote:
> On Fri, 2006-01-20 at 21:43 +0100, Thomas Wouters wrote:
> 
>> I don't believe this belongs in 2.4, since it can, actually, break code.
>> Code that depends on the current situation, _TestCase__attributename.
>> Fragile code, to be sure, but still. If there were a compelling reason to
>> backport, I guess it could be hacked to work right-ish, but subclassing
>> TestCase in this way, while convenient, isn't important enough to warrant
>> this (IMHO).
> 
> Exactly right.

You're right. Next time I'm going to decide in favor of not backporting.
I reverted both checkins, this and the urlparse one.


For 2.5, this would be a good opportunity to add additional schemes
to urlparse. Candidates?

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Phillip J. Eby]

At 11:19 AM 01/20/2006 -0800, Guido van Rossum wrote:
>(There *are*other uses besides the trampoline,
>right? :-)

It's easy to come up with use cases where you feed data *into* a generator 
(parsers and pipelines, for example).  I just don't know of any 
"simultaneous bidirectional" uses other than in association with a 
coroutine scheduler, or a dedicated pipelining tool.  In both the coroutine 
and pipelining cases, generator composition is an easy job for whatever 
coroutine or pipelining library is in use; it just happens external to the 
generator, perhaps by yielding a special value.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 343 and __context__()

[Nick Coghlan]

Jason Orendorff wrote:
> On 1/20/06, Nick Coghlan <[EMAIL PROTECTED]> wrote:
>> Jason Orendorff wrote:
>>> DecimalContext has a few problems.  In code where it matters, every
>>> function you write has to worry about it. (That is, you can't just
>>> write __decimal_context__ = ... at the top of the file and be done
>>> with it, the way you can with, say, __metaclass__.)
>> No, you write "decimal.setcontext(...)" instead.
> 
> You seem to be implying these are roughly equal in convenience; I
> disagree.  Suppose I have banking.py, in which it's important to use a
> particular precision and rounding.  Now I have to put context-munging
> code in every single function that banking.py exposes.  And around
> every 'yield'.  Even with 'with', that's a lot of extra lines of code.

Ah, I understand your point better now.

> I'd much prefer to put a one-liner at the top of the file, if it were
> possible (...but I don't see how, yet).

Me neither. The best I can come up with is a "precise" decorator that takes 
care of the context munging. Which would at least reduce the boilerplate to 
"@precise" on functions and "@precise_gen" on generators.

> Again, none of this is likely to matter--unless you're interleaving
> banking and heavy scientific calculations, which I try to avoid.  So,
> not a big deal.  Thanks for the response.

No worries. I agree there's still some awkwardness, but I do think it's an 
improvement on the status quo (where we have the same problems with 
interactions between modules and generators and thread-state, but dealing with 
them can't really be factored out at all).

>>>  And
>>> DecimalContext doesn't fit in with generators.
>> It does fit actually - you simply have to remember to restore the original
>> context around any invocations of yield.
> 
> Feh!  "Fit" is to "can be made to work with a bit of effort, just
> don't forget to follow the rules" as Python is to C++.

I see what you mean. I'm sure we can rely on PJE and others to push the limits 
of with statements and provide feedback on improving their interaction with 
generators in Python 2.6 :)

(although before that we need to bring mwh's patch up to speed with the svn 
trunk. . .)

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Nick Coghlan]

Phillip J. Eby wrote:
>> Thoughts?
> 
> If we have to have a syntax, "yield from sub_generator()" seems clearer 
> than "yieldthrough", and doesn't require a new keyword.

Andrew Koenig suggested the same phrasing last year [1], and I liked it then. 
I don't like it any more, though, as I think it is too inflexible, and we have 
a better option available (specifically, stealing "continue with an argument" 
from PEP 340). The following ramblings (try to) explain my reasoning :)

Guido does raise an interesting point. The introduction of "send" and "throw" 
means that the current simple loop approach does not easily allow values to be 
passed down to nested generators, nor does not it correctly terminate nested 
generators in response to an invocation of "throw". Because of the explicit 
for loop, the nested generator only gets cleaned up in response to GC - it 
never sees the exception that occurs in the body of the for loop (at the point 
of the yield expression).

The "yield from iterable" concept could be translated roughly as follows:

   itr = iter(iterable)
   try:
   send_input = itr.send  # Can input values be passed down?
   except AttributeError:
   send_input = None
   try:
   next = itr.next()  # Get the first output
   except StopIteration:
   pass
   else:
   while 1:
   try:
   input = yield next  # yield and get input
   except:
   try:
  throw_exc = itr.throw  # Can exception be passed down?
   except AttributeError:
   raise # Nope, reraise
   else:
   throw_exc(sys.exc_info()) # Yep, pass it down
   else:
   try:
   if send_input is None:
   if input is not None:
   raise TypeError("Cannot send input!")
   next = itr.next()
   else:
   next = send_input(input) # Pass input down
   except StopIteration:
   break

I'm not particularly happy with this, though, as not only is it horribly 
implicit and magical, it's trivial to accidentally break the chain - consider 
what happens if you naively do:
   yield from (x*x for x in sub_generator())

The chain has been broken - the sub generator no longer sees either passed in 
values or thrown exceptions, as the generator expression intercepts them 
without passing them down. Even worse, IMO, is that the syntax is entirely 
inflexible - we have no easy way to manipulate either the results sent from 
the generator, or the input values passed to it.

However, an idea from Guido's PEP 340 helps with the "send" part of the story, 
involving passing an argument to continue:

   def main_generator():
   ...
   for value in sub_generator():
   continue yield value

Here, sub_generator's "send" method would be invoked with the result of the 
call to yield value. Manipulation in either direction (input or output) is 
trivial:

   def main_generator():
   ...
   for value in sub_generator():
   input = yield value*value   # Square the output values
   continue input*input# Square the input values, too

You could even do odd things like yield each value twice, and then pass down 
pairs of inputs:

   def main_generator():
   ...
   for value in sub_generator():
   continue (yield value), (yield value)

The need to use a "continue" statement eliminates the temptation to use a 
generator expression, and makes it far less likely the downwards connection 
between the main generator and the sub generator will be accidentally broken.

Exception propagation is a different story. What do you want to propagate? All 
exceptions from the body of the for loop? Or just those from the yield 
statement?

Well, isn't factoring out exception processing part of what PEP 343 is for?

   # Simply make sure the generator is closed promptly
   def main_generator():
   ...
   with closing(sub_generator()) as gen:
   for value in gen:
   continue yield value

   # Or throw the real exception to the nested generator
   class throw_to(object):
   def __init__(self, gen):
   self.gen = gen
   def __enter__(self):
   return self.gen
   def __exit__(self, exc_type, *exc_details):
   if exc_type is not None:
   try:
   self.gen.throw(exc_type, *exc_details)
   except StopIteration:
   pass

   def main_generator():
   ...
   with throw_to(sub_generator()) as gen:
   for value in gen:
   continue yield value

   # We can even limit the propagated exceptions to those
   # from the outside world and leave the rest alone
   def main_generator():
   ...
   gen = sub_generator()
   for value in gen:
   with throw_to(gen):
   input = yield val

Re: [Python-Dev] yield back-and-forth?

[Nick Coghlan]

Nick Coghlan wrote:
> Exception propagation is a different story. What do you want to propagate? 
> All 
> exceptions from the body of the for loop? Or just those from the yield 
> statement?
> 
> Well, isn't factoring out exception processing part of what PEP 343 is for?

># We can even limit the propagated exceptions to those
># from the outside world and leave the rest alone
>def main_generator():
>...
>gen = sub_generator()
>for value in gen:
>with throw_to(gen):
>input = yield value
>continue input

A point I should have made here (but didn't, despite using it in a later 
example) is that the "with" versions don't allow the nested generator to 
suppress the exception.

A small refinement to gen.throw() that makes the first argument optional 
(similar to the raise statement itself) would make it straightforward to allow 
the generator to suppress passed in exceptions:

def main_generator():
...
gen = sub_generator()
for value in gen:
try:
input = yield value
except:
gen.throw() # Default to sys.exc_info(), just like raise
continue input

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Andrew Koenig]

> The discussion about PEP 343 reminds me of the following. Bram Cohen
> pointed out in private email that, before PEP 342, there wasn't a big
> need for a shortcut to pass control to a "sub-generator" because the
> following for-loop works well enough:

>  def main_generator():
>   ...
>   for value in sub_generator():
>   yield value

> but now that yield can return a value, that value might have to be
> passed into sub_generator(), not to mention of exceptions. I'm sure
> there's a way to write that (although I haven't found the time to
> figure it out) but I expect it to be cumbersome and subtle.

It looks to me like continuations are starting to rear their heads...




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield back-and-forth?

[Alex Martelli]

On Jan 20, 2006, at 1:39 PM, Phillip J. Eby wrote:

> At 11:19 AM 01/20/2006 -0800, Guido van Rossum wrote:
>> (There *are*other uses besides the trampoline,
>> right? :-)
>
> It's easy to come up with use cases where you feed data *into* a  
> generator
> (parsers and pipelines, for example).  I just don't know of any
> "simultaneous bidirectional" uses other than in association with a
> coroutine scheduler, or a dedicated pipelining tool.  In both the  
> coroutine

Hmm, what about an iterator of iterators, where the sub-iterators  
need to be advanced until some condition is satisfied, then processed  
from that point onwards (only those subiterators for which some item  
did satisfy the condition).  E.g.:

goodfiles = []
for afile in manyfiles:
   for aline in afile:
 if aline.startswith('*starthere*'):
   goodfiles.append(afile)
   break
for afile in goodfiles: ...

I'm sure we've all met this kind of issue (indeed, standard library  
module fileinput does support this case directly, by wrapping the  
double loop into one yielding lines, and still providing ways to get  
the current file and skipping to the next file).

It might be nice to wrap the general logic (double nested looping &c)  
in a generator leaving the specifics (exactly when to consider a  
subiterator 'good' and what to do in that case) out of it, sort of  
like fileinput does but with more generality than just files and  
lines thereof.  In Python 2.5 it seems that a .send call with a non- 
None argument might be a natural convention for the loop body to tell  
the generator "abandon this subiterator and yield it, then move to  
the next one".  E.g.:

goodfiles = []
allinput = inputfrom(manyfiles)
for aline in allinput:
   if aline.startswith('*starthere*'):
 goodfiles.append(allinput.send(True))
for afile in goodfiles: ...

Perhaps not the greatest of simplifications, but it might be  
generalized to (e.g.) recursive walks with the same client-logic as  
above, while the original (nested for-loops) is pretty rigid.  And  
the inputfrom generator doesn't look too complicated either:

def inputfrom(iterators):
   for iterator in iterators:
 for item in iterator:
   if (yield item) is not None:
 yield iterator
 break

Wouldn't this be considered a reasonable use for the new generator  
features?


Alex

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com