Re: [Python-Dev] doctest, unicode repr, and 2to3

[Nick Coghlan]

Glyph Lefkowitz wrote:
> 
> On Mar 4, 2010, at 11:30 PM, Barry Warsaw wrote:
> 
>> If you really want to test that it's a unicode, shouldn't you actually
>> test
>> its type?  (I'm not sure what would happen with that under 2to3.)
> 
> Presumably 2to3 will be smart enough to translate 'unicode' to 'str' and
> 'bytes' to... 'bytes'.  Just don't use 'str' in 2.x and you should be
> okay :).

In code (including doctest input lines) 2to3 can make those
translations, but I believe the doctest output lines are a different
story (since they're just arbitrary strings as far as the translator is
concerned).

I would expect 2to3 to be able to translate the following tests
correctly because the expected output stays the same even though the
commands change:
>>> print a6.headline
'Default headline'
>>> type(a6.headline) is type(u'')
True

But I don't see a ready way to support doctests where the expected
*output* changes between versions.

Cheers,
Nick.
-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Andrew Bennetts]

"Martin v. Löwis" wrote:
[...]
> Any proposal appreciated.

I propose screaming “help me, I have written a test suite using nothing
but string matching assertions, what is wrong with me?!”

-Andrew.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Calvin Spealman]

A young library solving an old problem in a way that conflicts with
many of the other implementations available for years and with zero
apparent users in the wild is not an appropriate candidate for a PEP.

On Fri, Mar 5, 2010 at 1:03 AM, Brian Quinlan  wrote:
> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it easier to
> execute Python functions asynchronously using threads and processes. It lets
> the user focus on their computational problem without having to build
> explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this group's
> feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
>    112272535095293,
>    112582705942171,
>    112272535095293,
>    115280095190773,
>    115797848077099,
>    1099726899285419]
>
> def is_prime(n):
>    if n % 2 == 0:
>        return False
>
>    sqrt_n = int(math.floor(math.sqrt(n)))
>    for i in range(3, sqrt_n + 1, 2):
>        if n % i == 0:
>            return False
>    return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
>    for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
>        print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and Fox
> News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
>    # Create a future for each URL load.
>    future_to_url = dict((executor.submit(load_url, url, 60), url)
>                         for url in URLS)
>
>    # Iterate over the futures in the order that they complete.
>    for future in futures.as_completed(future_to_url):
>        url = future_to_url[future]
>        if future.exception() is not None:
>            print('%r generated an exception: %s' % (url,
>                                                     future.exception()))
>        else:
>            print('%r page is %d bytes' % (url, len(future.result(
>
> Cheers,
> Brian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman  wrote:
> A young library solving an old problem in a way that conflicts with
> many of the other implementations available for years and with zero
> apparent users in the wild is not an appropriate candidate for a PEP.
>

Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package - but Brian's one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it's good
sugar, and is used by a fair number of people.

Let me also state - "my" vision of where futures would live would be
in a concurrent package - for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.

In any case; consider me a strong +1 to adding it.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Calvin Spealman]

I revert my objections. I still would like to see this in use "in the
wild" and I might even use it thusly, myself.

On Fri, Mar 5, 2010 at 9:50 AM, Jesse Noller  wrote:
> On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman  wrote:
>> A young library solving an old problem in a way that conflicts with
>> many of the other implementations available for years and with zero
>> apparent users in the wild is not an appropriate candidate for a PEP.
>>
>
> Baloney. A young library providing some syntactic sugar which uses
> primitives in the standard library to implement a common pattern is
> fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
> list prior to bringing it here. By the same argument, we should shunt
> all of the recent unittest changes and improvements into space, since
> golly, other people did it, why should we.
>
> This is something relatively simple, which I would gladly add in an
> instant to the multiprocessing package - but Brian's one-upped me in
> that regard and is providing something which works with both threads
> and processes handily. Take a look at multiprocessing.Pool for example
> - all that is some sugar on top of the primitives, but it's good
> sugar, and is used by a fair number of people.
>
> Let me also state - "my" vision of where futures would live would be
> in a concurrent package - for example:
>
> from concurrent import futures
>
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.
>
> For example, how many of us have sat down and implemented a thread
> pool on top of threading, I would hazard to say that most of us who
> use threading have done this, and probably more than once. It stands
> to reason that this is a common enough pattern to include in the
> standard library.
>
> In any case; consider me a strong +1 to adding it.
>
> jesse
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Nick Coghlan]

Jesse Noller wrote:
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.

Yes, I've often thought we should have a pool model that covers threads
as well as processes.

The reason I think "futures" works as a PEP and potential standard
library addition is that it is small enough to be readily reviewed and
maintained, and serves as a useful building block for more complex usage.

For a developer to get anything similar from a third party library is
almost certainly going to require buying into a much heavier framework.
A simple futures module provides a way to farm out worker tasks in a
standard fashion without having to build as much of your own
infrastructure every time.

I've read the various PEP checkins as they went by on the checkins list
- it gets a +0 from me (the only reason it isn't a +1 is because I
personally tend to write with a Thread+Queue style. However, I could
easily become a futures convert if they were readily available in the
standard library)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Antoine Pitrou]

Le Thu, 4 Mar 2010 23:30:12 -0500,
Barry Warsaw  a écrit :
> 
> If you really want to test that it's a unicode, shouldn't you
> actually test its type?  (I'm not sure what would happen with that
> under 2to3.)  Besides, the type of the string is very rarely
> important, so I think the u-prefix and quotes is mostly just noise.

String type is actually very important, if you don't want your
application/library to fail in the face of non-ASCII data.

That's why we did all this thing in py3k, after all :)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Michael Foord]

On 05/03/2010 15:56, Antoine Pitrou wrote:

Le Thu, 4 Mar 2010 23:30:12 -0500,
Barry Warsaw  a écrit :
   

If you really want to test that it's a unicode, shouldn't you
actually test its type?  (I'm not sure what would happen with that
under 2to3.)  Besides, the type of the string is very rarely
important, so I think the u-prefix and quotes is mostly just noise.
 

String type is actually very important, if you don't want your
application/library to fail in the face of non-ASCII data.

That's why we did all this thing in py3k, after all :)
   


Well, I'd like to see a 'unicode agnostic' mode for different reasons - 
although given that it all changes in Python 3 and it is almost 
certainly too late to get changes to doctest before the 2.7 release the 
point is probably moot.


IronPython has very different unicode behaviour to Python 2. IronPython 
is like Python 3 in this regard, all strings are unicode (str is 
unicode). This means that it rarely puts the u prefix on strings. So 
doctests that use string repr also fail on IronPython - the django 
doctest suite is a culprit in this regard too.


Dictionary iteration order is also different in IronPython - so dict 
reprs also fail. For dict reprs it would be nice if doctest compared in 
an order agnostic way. (Internally reconstruct the dict as a dict of 
strings from the repr and then compare using dict equality.)


I know it is easily *possible* to construct doctests that are robust 
against these differences, but the fact that the "obvious-thing" often 
fails in subtle ways on other platforms are *one* of the reasons I don't 
consider doctest to be a suitable tool for unit testing [1]. :-)


Michael

[1] It isn't the main reason and I realise that is entirely orthoganal 
anway... but doctest makes every line an assertion, and you have to jump 
through hoops if you don't want that behaviour. All besides the point. 
Doctest simply *rocks* for testing documentation examples - especially 
in conjunction with Sphinx.



Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Barry Warsaw]

On Mar 05, 2010, at 10:56 AM, Antoine Pitrou wrote:

>String type is actually very important, if you don't want your
>application/library to fail in the face of non-ASCII data.
>
>That's why we did all this thing in py3k, after all :)

That's not actually what I mean.  I meant that in doctests, you probably don't
need to be confronted by the string type every single time.  It's just not
that interesting for documentation, most of the time.  I have no problem
adding (unit)tests that ensure your outputs and values are of the expected
type though!

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Daniel Stutzbach]

On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:

> import futures
>

+1 on the idea, -1 on the name.  It's too similar to "from __future__ import
...".

Also, the PEP should probably link to the discussions on stdlib-sig?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
 wrote:
> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
>>
>> import futures
>
> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
> ...".

Futures is a common term for this, and implemented named this in other
languages. I don't think we should be adopting things that are common,
and found elsewhere and then renaming them.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[skip]

>>> import futures
>> 
>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ 
import
>> ...".

Jesse> Futures is a common term for this, and implemented named this in
Jesse> other languages. I don't think we should be adopting things that
Jesse> are common, and found elsewhere and then renaming them.

Perhaps, but is it a common term for Python programmers (or the target
population for Python)?  I've never heard of it.  "futures" to me are
futures contracts in a trading environment (that's the industry I work in).
No matter how well known the term is in the environment where it's used
today you have to be sensitive to other meanings of the term.

Skip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 11:56 AM,   wrote:
>    >>> import futures
>    >>
>    >> +1 on the idea, -1 on the name.  It's too similar to "from __future__ 
> import
>    >> ...".
>
>    Jesse> Futures is a common term for this, and implemented named this in
>    Jesse> other languages. I don't think we should be adopting things that
>    Jesse> are common, and found elsewhere and then renaming them.
>
> Perhaps, but is it a common term for Python programmers (or the target
> population for Python)?  I've never heard of it.  "futures" to me are
> futures contracts in a trading environment (that's the industry I work in).
> No matter how well known the term is in the environment where it's used
> today you have to be sensitive to other meanings of the term.
>

It's a common programming term. I don't think we should make a new
name - I mean, how many different names for green threads / coroutines
or "flavors" of those concepts are out there because someone painted
the bike shed a different color? I mean, there's prior art here:

http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Curt Hagenlocher]

On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller  wrote:
>
> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>  wrote:
> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
> >>
> >> import futures
> >
> > +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
> > ...".
>
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

Another common term for this is a "promise".

--
Curt Hagenlocher
c...@hagenlocher.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Daniel Stutzbach]

On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller  wrote:

> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>

 According to that link, Java has a module named "Concurrent" with an
interface named "Future".  You're proposing a module named "Futures" with a
class named "Future".

Why not name your module "concurrent"?  That would eliminate the confusion
with "from __future__".  I don't see a problem with keeping the class name.

Plus, a "concurrent" module might be useful for things other than Futures,
in the future. ;-)

Just my 0.02 cents,
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[skip]

Jesse> 
http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

Without reading that I can assure you that not everybody has drunk the Java
Kool-Aid.  Just because Sun thought it was a fine term doesn't mean everyone
else will.  I've been a professional programmer for about 30 years, have
never touched Java, and the only times I've seen its effect in Python I have
been unimpressed (logging and unittest).

Google for "future".  The Java doc is aways down on the front page.  Now
Google for "futures". It's nowhere to be found.  As far as I can tell, all
but one hit on the first page are for my definition of "futures".  At the
bottom of the page the "searches related to _futures_" are all related to
trading futures or other derivatives except one.  That appears to be related
to the LPGA futures tour.  Java and its definition of the term is nowhere to
be found.

I suppose you can argue that it should be called "future".  I still contend
that the name is *not* intuitive, no matter how well known it happens to be
within the Java world.  Don't name it "futures" though.  That has nothing to
do with common definitions of the term.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
 wrote:
> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller  wrote:
>>
>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>
>  According to that link, Java has a module named "Concurrent" with an
> interface named "Future".  You're proposing a module named "Futures" with a
> class named "Future".
>
> Why not name your module "concurrent"?  That would eliminate the confusion
> with "from __future__".  I don't see a problem with keeping the class name.
>
> Plus, a "concurrent" module might be useful for things other than Futures,
> in the future. ;-)
>

Brian's module is named futures; I am +1'ing his proposal, and also
suggesting we put it under concurrent/ package name. This means you
would do the following:

from concurrent import futures

and in the future:
from concurrent import pool

And so on.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[exarkun]

On 05:06 pm, c...@hagenlocher.org wrote:

On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller  wrote:


On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
 wrote:
> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  
wrote:

>>
>> import futures
>
> +1 on the idea, -1 on the name.  It's too similar to "from 
__future__ import

> ...".

Futures is a common term for this, and implemented named this in other
languages. I don't think we should be adopting things that are common,
and found elsewhere and then renaming them.


Another common term for this is a "promise".


Promises aren't exactly the same.  This would be a particularly bad name 
to apply here.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Guido van Rossum]

On Fri, Mar 5, 2010 at 9:55 AM, Jesse Noller  wrote:
> On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
>  wrote:
>> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller  wrote:
>>>
>>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>>
>>  According to that link, Java has a module named "Concurrent" with an
>> interface named "Future".  You're proposing a module named "Futures" with a
>> class named "Future".
>>
>> Why not name your module "concurrent"?  That would eliminate the confusion
>> with "from __future__".  I don't see a problem with keeping the class name.
>>
>> Plus, a "concurrent" module might be useful for things other than Futures,
>> in the future. ;-)
>>
>
> Brian's module is named futures; I am +1'ing his proposal, and also
> suggesting we put it under concurrent/ package name. This means you
> would do the following:
>
> from concurrent import futures
>
> and in the future:
> from concurrent import pool
>
> And so on.

"Future" is a pretty standard CS term for this concept (as noted
"promise" is another), and it wasn't invented by Java. I am not
worried at all about confusion with __future__ (which is after all
surrounded by dunders and only appears in one specific context).

FWIW, there are two different ways of designing the API for such a
concept. In one, the "future" or "promise" object masquerades as an
instance of the eventual result, like a proxy, and accessing any
attribute would block until the result is available. I don't like this
design much; the masquerading is never perfect, and attribute accesses
become risks of exceptions related to the promised work. In the other
design, which this PEP proposes, there is an explicit call to get the
value, which blocks as needed.

I found a wikipedia article
(http://en.wikipedia.org/wiki/Futures_and_promises#Implicit_vs_explicit)
that calls the former "implicit" and the latter "explicit". According
to this article, the term "promise" was coined slightly older (by a
year or so). I have no strong preference for either term; I suspect
that if this were the only objection to the PEP Brian would be happy
to rename futures to promises.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Bill Janssen]

s...@pobox.com wrote:

> 
> Jesse> 
> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
> 
> Without reading that I can assure you that not everybody has drunk the Java
> Kool-Aid.  Just because Sun thought it was a fine term doesn't mean everyone
> else will.  I've been a professional programmer for about 30 years, have
> never touched Java, and the only times I've seen its effect in Python I have
> been unimpressed (logging and unittest).
> 
> Google for "future".  The Java doc is aways down on the front page.  Now
> Google for "futures". It's nowhere to be found.  As far as I can tell, all
> but one hit on the first page are for my definition of "futures".  At the
> bottom of the page the "searches related to _futures_" are all related to
> trading futures or other derivatives except one.  That appears to be related
> to the LPGA futures tour.  Java and its definition of the term is nowhere to
> be found.

This term, the "future", has been in use at least since 1985, even if
the Web hasn't, and even if Google can't seem to separate this use from
drivel about stock trading.  See Halstead's 1985 TOPLAS paper on
MultiLisp.

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.97.1841

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Bill Janssen]

http://home.pipeline.com/~hbaker1/Futures.html (1977)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] argparse ugliness

[Neal Becker]

I generally enjoy argparse, but one thing I find rather
ugly and unpythonic.

parser.add_argument ('--plot', action='store_true')

Specifying the argument 'action' as a string is IMO ugly.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Guido van Rossum]

On Fri, Mar 5, 2010 at 10:30 AM,   wrote:
> On 05:06 pm, c...@hagenlocher.org wrote:
>>
>> On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller  wrote:
>>>
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>>  wrote:
>>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan 
>>> > wrote:
>>> >>
>>> >> import futures
>>> >
>>> > +1 on the idea, -1 on the name.  It's too similar to "from __future__
>>> > import
>>> > ...".
>>>
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>>
>> Another common term for this is a "promise".
>
> Promises aren't exactly the same.  This would be a particularly bad name to
> apply here.

Please explain. Even the Wikipedia article
(http://en.wikipedia.org/wiki/Futures_and_promises), despite promising
to explain the difference, didn't explain it.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ugliness

[Brian Curtin]

On Fri, Mar 5, 2010 at 12:51, Neal Becker  wrote:

> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
>
>parser.add_argument ('--plot', action='store_true')
>
> Specifying the argument 'action' as a string is IMO ugly.
>

What else would you propose?
FWIW, this is the same in optparse.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Terry Reedy]

On 3/4/2010 11:11 PM, "Martin v. Löwis" wrote:

Johan Harjano ran into an interesting problem when trying to run the
Django test suite under Python 3.1.

Django has doctests of the form


a6.headline

u'Default headline'

Even when converting the doctest with 2to3, the expected output is
unmodified. However, in 3.x, the expected output will change (i.e. not
produce an u"" prefix anymore).

Now, it might be possible to reformulate the test case (e.g. use print()
instead of relying on repr), however, this is undesirable as a) the test
should continue to test in 2.x that the result object is a unicode
string, and b) it makes the test less readable.

I would like to find a solution where this gets automatically corrected,
e.g. through 2to3, or through changes to doctest, or through changes of
str.__repr__.

Any proposal appreciated.


What is the easiest thing that works?

If 2to3 can fix string literals and fix code within doc strings, would 
it be difficult to fix expected strings within doc strings?


On the otherhand, as Foord pointed out, the 'u' prefix is something of a 
CPythonism not required by the language def, so an 'ignore leading u on 
expected output' flag would be useful. But 'correcting'


>>> a,b
u'ah', u'hah'

is too much to expect. Doctest is both useful and fragile because it 
*almost* imitates human testing methods.


If tests like the above were bunched, and my primary goal were to make 
Django tests work, and work now, without changing the style, I would be 
tempted to use something like the following:


def u(s): #import as needed from a testutil module
  if type(s) is unicode: # str in 3.x
print(s)
  else print('') #or any 'nevermatch' string

>>> u(a6.headline)
'Default headline'

which both retains the type test and readability. This would be a 
project specific solution, but then, the other comments suggest that 
wanting to combine a type and value test this way seems to be someone 
project specific.


Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Martin v. Löwis]

Andrew Bennetts wrote:
> "Martin v. Löwis" wrote:
> [...]
>> Any proposal appreciated.
> 
> I propose screaming “help me, I have written a test suite using nothing
> but string matching assertions, what is wrong with me?!”

Thanks a lot.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[exarkun]

On 07:10 pm, gu...@python.org wrote:

On Fri, Mar 5, 2010 at 10:30 AM,   wrote:

On 05:06 pm, c...@hagenlocher.org wrote:


On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller  
wrote:


On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
 wrote:
> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan 


> wrote:
>>
>> import futures
>
> +1 on the idea, -1 on the name.  It's too similar to "from 
__future__

> import
> ...".

Futures is a common term for this, and implemented named this in 
other
languages. I don't think we should be adopting things that are 
common,

and found elsewhere and then renaming them.


Another common term for this is a "promise".


Promises aren't exactly the same.  This would be a particularly bad 
name to

apply here.


Please explain. Even the Wikipedia article
(http://en.wikipedia.org/wiki/Futures_and_promises), despite promising
to explain the difference, didn't explain it.


The "explicit" futures on the wikipedia page seems to cover what is 
commonly referred to as a future.  For example, Java's futures look like 
this.


The "implicit" futures are what is generally called a promise.  For 
example, E's promises look like this.


Though the difference is mainly one of API, it turns out to make a 
significant difference in what you can accomplish.  Promises are much 
more amenable to the pipelining optimization, for example.  They're also 
much harder to implement in Python without core language changes.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Brett Cannon]

On Fri, Mar 5, 2010 at 09:55, Jesse Noller  wrote:

> On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
>  wrote:
> > On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller  wrote:
> >>
> >> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
> >
> >  According to that link, Java has a module named "Concurrent" with an
> > interface named "Future".  You're proposing a module named "Futures" with
> a
> > class named "Future".
> >
> > Why not name your module "concurrent"?  That would eliminate the
> confusion
> > with "from __future__".  I don't see a problem with keeping the class
> name.
> >
> > Plus, a "concurrent" module might be useful for things other than
> Futures,
> > in the future. ;-)
> >
>
> Brian's module is named futures; I am +1'ing his proposal, and also
> suggesting we put it under concurrent/ package name. This means you
> would do the following:
>
> from concurrent import futures
>
> and in the future:
> from concurrent import pool
>
> And so on.



So I don't quite get what you are after here. Are you wanting to eventually
have a generic pool class that you can simply import and use that is always
set to the best option for the platform?

And as for moving stuff from multiprocessing into the concurrent namespace,
are you thinking like concurrent.multiprocessing? I guess I am just trying
to figure out what the abstraction is you are after in the package
namespace.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jesse Noller wrote:
> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>  wrote:
>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
>>> import futures
>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>> ...".
> 
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

- -1 to the name from me as well:  it isn't "scoped" properly to make it
clear what the module is about.  If they were inside a pacakge named
'concurrency' or some such (as hinted by Jesse Noller, I think), the
clash would go away.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRapAACgkQ+gerLs4ltQ7dBwCfRmMuq6X9VE8usYgSScXEA1D0
1PsAoI8MR5hjPjq9C7MFPTZhcO/T+NM4
=7wpK
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Brett Cannon]

The PEP says that futures.wait() should only use keyword arguments past its
first positional argument, but the PEP has the function signature as
``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.  Should it be
``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?

On Thu, Mar 4, 2010 at 22:03, Brian Quinlan  wrote:

> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it easier to
> execute Python functions asynchronously using threads and processes. It lets
> the user focus on their computational problem without having to build
> explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this group's
> feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
>112272535095293,
>112582705942171,
>112272535095293,
>115280095190773,
>115797848077099,
>1099726899285419]
>
> def is_prime(n):
>if n % 2 == 0:
>return False
>
>sqrt_n = int(math.floor(math.sqrt(n)))
>for i in range(3, sqrt_n + 1, 2):
>if n % i == 0:
>return False
>return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
>for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
>print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and Fox
> News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
>'http://www.cnn.com/',
>'http://europe.wsj.com/',
>'http://www.bbc.co.uk/',
>'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
># Create a future for each URL load.
>future_to_url = dict((executor.submit(load_url, url, 60), url)
> for url in URLS)
>
># Iterate over the futures in the order that they complete.
>for future in futures.as_completed(future_to_url):
>url = future_to_url[future]
>if future.exception() is not None:
>print('%r generated an exception: %s' % (url,
> future.exception()))
>else:
>print('%r page is %d bytes' % (url, len(future.result(
>
> Cheers,
> Brian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 3:31 PM, Brett Cannon  wrote:

>
> So I don't quite get what you are after here. Are you wanting to eventually
> have a generic pool class that you can simply import and use that is always
> set to the best option for the platform?
> And as for moving stuff from multiprocessing into the concurrent namespace,
> are you thinking like concurrent.multiprocessing? I guess I am just trying
> to figure out what the abstraction is you are after in the package
> namespace.
> -Brett

My goal would be to put futures into a "concurrent" package - as it is
an abstraction that allows for threads, and processes to be used. By
default; I don't think we'd make a guess based on the platform, but
rather pick a sane default (such as threads).

After that; I would begin to remove chunks of multiprocessing (such as
pool) and adapt it to the same type of "pick threads or processes"
that futures uses. For example:

from concurrent import Pool

 x = Pool(4, worker_primitive=Thread())

And so on. The end-goal would be to make concurrent.* a package
containing common abstractions/patterns which can use threads or
processes interchangeably.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Guido van Rossum]

On Fri, Mar 5, 2010 at 12:18 PM,   wrote:
> The "explicit" futures on the wikipedia page seems to cover what is commonly
> referred to as a future.  For example, Java's futures look like this.
>
> The "implicit" futures are what is generally called a promise.  For example,
> E's promises look like this.

Fair enough, though the article confuses the matter by using the words
more or less interchangeably.

> Though the difference is mainly one of API, it turns out to make a
> significant difference in what you can accomplish.  Promises are much more
> amenable to the pipelining optimization, for example.  They're also much
> harder to implement in Python without core language changes.

That's why implicit futures (by any name) aren't on the table.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Guido van Rossum]

On Thu, Mar 4, 2010 at 8:11 PM, "Martin v. Löwis"  wrote:
> Johan Harjano ran into an interesting problem when trying to run the
> Django test suite under Python 3.1.
>
> Django has doctests of the form
>
 a6.headline
> u'Default headline'
>
> Even when converting the doctest with 2to3, the expected output is
> unmodified. However, in 3.x, the expected output will change (i.e. not
> produce an u"" prefix anymore).
>
> Now, it might be possible to reformulate the test case (e.g. use print()
> instead of relying on repr), however, this is undesirable as a) the test
> should continue to test in 2.x that the result object is a unicode
> string, and b) it makes the test less readable.
>
> I would like to find a solution where this gets automatically corrected,
> e.g. through 2to3, or through changes to doctest, or through changes of
> str.__repr__.
>
> Any proposal appreciated.

How about a heuristic rule (which you have to explicitly select) that
changes u'XXX' into 'XXX' inside triply-quoted strings given certain
contexts, e.g. only at the start of the line, only if there is a
nearby preceding line starting with '>>>'? Exactly what context is of
the right strength will have to be determined experimentally; if there
are a lot of tests outputting things like [u'...'] or {u'...': u'...'}
the context may have to be made more liberal. Possibly \bu('.*'|".*")
would do it?

The issue shows (yet again) a general problem with doctests being
overspecified -- the test shouldn't care that the output starts with
'u', it should only care that the value is unicode, but there's no
easy way to express this in doctests. But since these doctests exist I
suggest that the practical way forward is to stick with them rather
than trying to reformulate all the tests.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Martin v. Löwis]

> The issue shows (yet again) a general problem with doctests being
> overspecified -- the test shouldn't care that the output starts with
> 'u', it should only care that the value is unicode, but there's no
> easy way to express this in doctests. But since these doctests exist I
> suggest that the practical way forward is to stick with them rather
> than trying to reformulate all the tests.

Ok, I think I would prefer that. One approach I considered was to
override sys.displayhook, so that I can determine what parts of the
output are "single" repr results. Of course, this wouldn't catch repr
strings that were ultimately print()ed (not sure whether that
restriction would affect Django).

The other (more aggressive) approach is the heuristics you propose,
which may end up with false negatives. doctest already has a wildcard
matching flag, so it would grow another one.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jesse Noller]

On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Jesse Noller wrote:
>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>  wrote:
>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
 import futures
>>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>>> ...".
>>
>> Futures is a common term for this, and implemented named this in other
>> languages. I don't think we should be adopting things that are common,
>> and found elsewhere and then renaming them.
>
> - -1 to the name from me as well:  it isn't "scoped" properly to make it
> clear what the module is about.  If they were inside a pacakge named
> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
> clash would go away.

If people agree with this; do you feel the proposal of said namespace
should be a separate PEP, or piggy back on this? I don't want to piggy
back on Brian's hard work.

Jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Georg Brandl]

Am 05.03.2010 20:37, schrieb Terry Reedy:
> On 3/4/2010 11:11 PM, "Martin v. Löwis" wrote:
>> Johan Harjano ran into an interesting problem when trying to run the
>> Django test suite under Python 3.1.
>>
>> Django has doctests of the form
>>
> a6.headline
>> u'Default headline'
>>
>> Even when converting the doctest with 2to3, the expected output is
>> unmodified. However, in 3.x, the expected output will change (i.e. not
>> produce an u"" prefix anymore).
>>
>> Now, it might be possible to reformulate the test case (e.g. use print()
>> instead of relying on repr), however, this is undesirable as a) the test
>> should continue to test in 2.x that the result object is a unicode
>> string, and b) it makes the test less readable.
>>
>> I would like to find a solution where this gets automatically corrected,
>> e.g. through 2to3, or through changes to doctest, or through changes of
>> str.__repr__.
>>
>> Any proposal appreciated.
> 
> What is the easiest thing that works?
> 
> If 2to3 can fix string literals and fix code within doc strings, would 
> it be difficult to fix expected strings within doc strings?

Yes.  Expected output from doctests can be anything.  Doing conversions
on it that are correct only for Python code is potentially producing many
false positives.  Heuristics need to be applied that can get very intricate.

> On the otherhand, as Foord pointed out,

Prefect?

> the 'u' prefix is something of a 
> CPythonism not required by the language def, so an 'ignore leading u on 
> expected output' flag would be useful. But 'correcting'
> 
>  >>> a,b
> u'ah', u'hah'
> 
> is too much to expect. Doctest is both useful and fragile because it 
> *almost* imitates human testing methods.

Yes, it is documentation and tests at the same time :)

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Guido van Rossum]

On Fri, Mar 5, 2010 at 1:42 PM, Jesse Noller  wrote:
> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>>  wrote:
 On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
> import futures
 +1 on the idea, -1 on the name.  It's too similar to "from __future__ 
 import
 ...".
>>>
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>>
>> - -1 to the name from me as well:  it isn't "scoped" properly to make it
>> clear what the module is about.  If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
>
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

A simple renaming of futures to concurrency.futures seems easy enough
to swallow. (Though I haven't kept track of what other modules the PEP
proposes.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Desired changes to Hg emails to python-checkins

[Georg Brandl]

Am 04.03.2010 21:20, schrieb Brett Cannon:
> 1) I miss not having the affected files listed in the subject line.

I actually like the (first line of the) commit message better.  It
communicates much better what the change is about (and for me, if I
want to look at/review it).

However, what I miss is the summary of affected files at the top of
the message.  Having to scroll down all of the diffs to find out if e.g.
a change to a documentation file is included is tedious, even if the
diffs are colored like they are for me.

> 2) The To field is set to h...@python.org  which
> gets rejected as an invalid email address if you reply. Would be better
> to set it to python-checkins so that the habitual reply to a checkin
> won't get rejected.

+1.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Antoine Pitrou]

Le Fri, 5 Mar 2010 17:03:02 +1100,
Brian Quinlan  a écrit :
> 
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/

Ok, here is my take on it:

> cancel()
> 
> Attempt to cancel the call. If the call is currently being executed
> then it cannot be cancelled and the method will return False,
> otherwise the call will be cancelled and the method will return
> True.

I think it shouldn't return anything, and raise an exception if
cancelling failed. It is really an error condition, and ignoring the
result doesn't seem right.

> Future.running()
> 
> Return True if the call is currently being executed and cannot be
> cancelled.
> 
> Future.done()
> 
> Return True if the call was successfully cancelled or finished
> running.

These don't really make sense since the future is executing
concurrently. By the time the result is returned, it can already be
wrong. I advocate removing those two methods.

> The following Future methods are meant for use in unit tests and
> Executor implementations.

Their names should then be preceded by an underscore '_'. We don't want
people to think they are public APIs and start relying on them.

> wait(fs, timeout=None, return_when=ALL_COMPLETED)
> [...]
> 
> This method should always be called using keyword arguments

I don't think this is right. Keyword arguments are nice, but mandating
them too often is IMO a nuisance (after all, it makes things longer to
type and requires you to remember the exact parameter names).
Especially when the method only takes at most 3 arguments.

IMO, keyword-only arguments are mostly useful when there are a lot of
positional arguments before, and you want to help the user use the
right calling signature.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] doctest, unicode repr, and 2to3

[Žiga Seilnacht]

Martin v. Löwis wrote:

Johan Harjano ran into an interesting problem when trying to run the
Django test suite under Python 3.1.

Django has doctests of the form

>>> a6.headline
u'Default headline'

Even when converting the doctest with 2to3, the expected output is
unmodified. However, in 3.x, the expected output will change (i.e. not
produce an u"" prefix anymore).

Now, it might be possible to reformulate the test case (e.g. use print()
instead of relying on repr), however, this is undesirable as a) the test
should continue to test in 2.x that the result object is a unicode
string, and b) it makes the test less readable.

I would like to find a solution where this gets automatically corrected,
e.g. through 2to3, or through changes to doctest, or through changes of
str.__repr__.

Any proposal appreciated.


You can use a custom DocTestRunner that replaces sys.displayhook in
its run() method and records the changed output. Something like the
attached seems to do the trick.

Regards,
Ziga

import sys
import doctest
import linecache
import __builtin__


def greet():
"""
The standard greeting, in unicode.

>>> greet()
u'Hello, world!'
"""
return u'Hello, world!'


orig_displayhook = sys.displayhook

def py3_displayhook(value):
if not isinstance(value, unicode):
return orig_displayhook(value)
__builtin__._ = value
s = repr(value)
if s.startswith(("u'", 'u"')):
s = s[1:]
print >> sys.stdout, s


class Runner(doctest.DocTestRunner):

converted_files = {}

def __init__(self, checker=None, verbose=None, optionflags=0):
doctest.DocTestRunner.__init__(self, checker, False, optionflags)


def run(self, test, compileflags=None, out=None, clear_globs=True):
fn = test.filename
if fn not in self.converted_files:
self.converted_files[fn] = linecache.getlines(fn)
sys.displayhook = py3_displayhook
try:
return doctest.DocTestRunner.run(self, test, compileflags,
  out, clear_globs)
finally:
sys.displayhook = orig_displayhook


def report_failure(self, out, test, example, got):
lines = [" " * example.indent + line for line in got.splitlines(True)]
pos = test.lineno + example.lineno + 1
self.converted_files[test.filename][pos:pos+len(lines)] = lines




def _test():
import __main__
finder = doctest.DocTestFinder()
runner = Runner()
for test in finder.find(__main__):
runner.run(test)
print "".join(runner.converted_files[__main__.__file__])


if __name__ == "__main__":
_test()
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jesse Noller wrote:
> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>>  wrote:
 On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan  wrote:
> import futures
 +1 on the idea, -1 on the name.  It's too similar to "from __future__ 
 import
 ...".
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>> - -1 to the name from me as well:  it isn't "scoped" properly to make it
>> clear what the module is about.  If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
> 
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

I'm just expressiong a preference for scoping the name, and don't want
to preempt the process.  If your proposed work on factoring common stuff
out of multiprocessing would sit in the same conceptual space, then
sharing the package name seems like a good plan to me.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRj74ACgkQ+gerLs4ltQ5RNACeKYku88A9PBuQR46QTl7GrEwo
mPEAoLdYyi+TLGYFw4SRAIM8zBsNvwxr
=iPkb
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ugliness

[Steven Bethard]

On Fri, Mar 5, 2010 at 10:51 AM, Neal Becker  wrote:
> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
>
>    parser.add_argument ('--plot', action='store_true')
>
> Specifying the argument 'action' as a string is IMO ugly.

If it really bothers you, you can use:

action=argparse._StoreTrueAction

instead. Undocumented (purposefully) but that's all argparse is doing
under the covers.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou  wrote:
> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan  a écrit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

The caller can't avoid the error here by querying the future, because
of the problem you point out below, so I'm inclined to think that "the
future was already started" should be a return value rather than an
exception (although that may be my C++ background showing through).
Would calling the method try_cancel() work better?

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

"done()" can only be wrong in one direction though. If it returns
True, it'll never again return False, which can be useful (although
perhaps only for polling, which might be an argument for removing it
anyway).

"running()" becomes True and then False again, so I agree with your
objection. A "started()" function would only go from False to True
once. Maybe that's a better function?

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ugliness

[Antoine Pitrou]

Le Fri, 05 Mar 2010 13:51:15 -0500,
Neal Becker  a écrit :
> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
> 
> parser.add_argument ('--plot', action='store_true')

I would argue that a string is actually more Pythonic than
integers or anonymous objects repurposed as magic constants.
(I'm looking at things such as SEEK_SET and friends)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Stephen J. Turnbull]

Guido van Rossum writes:

 > "Future" is a pretty standard CS term for this concept (as noted
 > "promise" is another),

I like the term "promise" better.  "Future" is very generic ("not now,
but later"), whereas a "promise" is something I don't get from you
now, but you will give me later.

The wikipedia article is not very helpful on the implicit vs. explicit
distinction.  As far as I can tell from it, that distinction isn't
really attached to "future" vs "promise."  The only distinction the
article described was in the context of the Alice language, where a
future = promise (read-only) plus resolver (mutator).  IMO that's not
a compelling reason for adopting "future" in Python.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Stephen J. Turnbull]

exar...@twistedmatrix.com writes:

 > The "explicit" futures on the wikipedia page seems to cover what is 
 > commonly referred to as a future.  For example, Java's futures look like 
 > this.
 > 
 > The "implicit" futures are what is generally called a promise.  For 
 > example, E's promises look like this.

*sigh*  All I can say is "it's a damned shame that there are no native
speakers of English working in computer science."

I have to admit Jean-Paul's explanation a pretty convincing reason for
adopting "future" rather than "promise".  But I'm with Skip, I would
prefer that the module be named "future" rather than "futures".
(Especially when wearing my Professional Economist sweatshirt. :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Phillip J. Eby]

At 01:03 AM 3/5/2010, Brian Quinlan wrote:

Hi all,

I recently submitted a daft PEP for a package designed to make it
easier to execute Python functions asynchronously using threads and
processes. It lets the user focus on their computational problem
without having to build explicit thread/process pools and work queues.

The package has been discussed on stdlib-sig but now I'd like this
group's feedback.


My immediate reaction is that this would be a lot more useful if it 
built on an API for coroutine yielding/interaction, similar to what's 
in say, Eventlet.  That would seem to make it easier to write 
synchronous-looking code that operates on futures, and allow futures 
to be composed more cleanly.


ISTM that if futures were standardized before a coroutine API, it 
would lead to effective orphaning ala what happened with asyncore, 
especially since the new library is, well, new.


I'm somewhat concerned that, as described, the proposed API adds 
little over what's relatively easy to do with a mature coroutine 
framework like Eventlet, while at the same time creating yet another 
alternative (and mutually incompatible) event loop system in the 
stdlib, beyond the ones that are already in asyncore, tkinter, and 
the various SocketServer subclasses.


As far as naming goes, Twisted uses the term "Deferred" for this 
concept (and also has a very mature API for handling them).


And speaking of Twisted, it seems to me that the PEP would be much 
improved in general by learning from some of the lessons of other 
systems.  I don't think that Java's example is really the best one to 
follow in this instance, compared to the many existing async 
frameworks that have Python-specific experience and APIs to learn from.


Or, to put it another way, something that worries me about this PEP 
is that nearly all of its Python-related citations are for 
*discussions* of futures, with the only previous Python 
implementation cited being a crude sketch of a cookbook recipe.  The 
PEP also doesn't address questions of interoperability with existing 
solutions, compare features with them, or even so much as say, "There 
are other production implementations of this concept in Python, but 
we are going to pretend they don't exist."  ;-)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby  wrote:
> I'm somewhat concerned that, as described, the proposed API ... [creates] yet 
> another alternative (and
> mutually incompatible) event loop system in the stdlib ...

Futures are a blocking construct; they don't involve an event loop.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Phillip J. Eby]

At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:

On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby  wrote:
> I'm somewhat concerned that, as described, the proposed API ... 
[creates] yet another alternative (and

> mutually incompatible) event loop system in the stdlib ...

Futures are a blocking construct; they don't involve an event loop.


And where they block is in a loop, waiting for events (completed 
promises) coming back from other threads or processes.


The Motivation section of the PEP also stresses avoiding reinvention 
of such loops, and points to the complication of using more than one 
at a time as a justification for the mechanism.  It seems relevant to 
at least address why wrapping multiprocessing and multithreading is 
appropriate, but *not* dealing with any other form of sync/async 
boundary, *or* composition of futures.


On which subject, I might add, the PEP is silent on whether executors 
are reentrant to the called code.  That is, can I call a piece of 
code that uses futures, using the futures API?  How will the called 
code know what executor to use?  Must I pass it one explicitly?  Will 
that work across threads and processes, without explicit support from the API?


IOW, as far as I can tell from the PEP, it doesn't look like you can 
compose futures without *global* knowledge of the application...  and 
in and of itself, this seems to negate the PEP's own motivation to 
prevent duplication of parallel execution handling!


That is, if I use code from module A and module B that both want to 
invoke tasks asynchronously, and I want to invoke A and B 
asynchronously, what happens?  Based on the design of the API, it 
appears there is nothing you can do except refactor A and B to take 
an executor in a parameter, instead of creating their own.


It seems therefore to me that either the proposal does not define its 
scope/motivation very well, or it is not well-equipped to address the 
problem it's setting out to solve.  If it's meant to be something 
less ambitious -- more like a recipe or example -- it should properly 
motivate that scope.  If it's intended to be a robust tool for 
composing different pieces of code, OTOH, it should absolutely 
address the issue of writing composable code...  since, that seems to 
be what it says the purpose of the API is.  (I.e., composing code to 
use a common waiting loop.)


And, existing Python async APIs (such as Twisted's Deferreds) 
actually *address* this issue of composition; the PEP does 
not.  Hence my comments about not looking at existing implementations 
for API and implementation guidance.  (With respect to what the API 
needs, and how it needs to do it, not necessarily directly copying 
actual APIs or implementations.  Certainly some of the Deferred API 
naming has a rather, um, "twisted" vocabulary.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Jeffrey Yasskin]

On Fri, Mar 5, 2010 at 9:47 PM, Stephen J. Turnbull  wrote:
> Guido van Rossum writes:
>
>  > "Future" is a pretty standard CS term for this concept (as noted
>  > "promise" is another),
>
> I like the term "promise" better.  "Future" is very generic ("not now,
> but later"), whereas a "promise" is something I don't get from you
> now, but you will give me later.
>
> The wikipedia article is not very helpful on the implicit vs. explicit
> distinction.  As far as I can tell from it, that distinction isn't
> really attached to "future" vs "promise."  The only distinction the
> article described was in the context of the Alice language, where a
> future = promise (read-only) plus resolver (mutator).  IMO that's not
> a compelling reason for adopting "future" in Python.

It seems like a good idea to follow the choice other languages have
used for the name (if they tend to agree) regardless of whether the
evil Java followed it too. So let's take a poll:

Io: Uses "future" to refer to the implicit kind
(http://www.iolanguage.com/scm/io/docs/IoGuide.html#Concurrency-Futures)
Alice ML: Uses "future" to refer to the implicit kind, and "promise"
to refer to a handle that can fill in the future
(http://www.ps.uni-saarland.de/alice/manual/futures.html)
Java: Uses "future" to refer to the explicit kind.
(http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html)
AmbientTalk: Uses "future" to refer to something more like a deferred:
you register callbacks to run when the future is resolved.
(http://soft.vub.ac.be/amop/at/tutorial/actors#futures)
C++0x: Uses "future" to refer to the explicit kind; "promise"
similarly to AliceML, and "packaged_task" to get a future from a
callable. (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3035.pdf
section 30.6)
E: Uses "promise" to refer to the implicit kind, and "resolver" to
refer to a handle that can fill in the promise.
(http://wiki.erights.org/wiki/Promise)
Oz: A "future" is a read-only logic variable. I'm not entirely sure
what that means.
(http://www.mozart-oz.org/documentation/dstutorial/node2.html#label61)
PLT Scheme: Uses "future" to refer to the explicit kind.
(http://docs.plt-scheme.org/futures/index.html)
C#: Uses "Task" to refer to the explicit kind.
(http://msdn.microsoft.com/en-us/library/dd321424(VS.100).aspx)
Id, from 1991: Used "I-structure" to refer to the implicit kind.
(http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4920)
Scala: Uses "Future" to refer to the explicit kind.
(http://www.scala-lang.org/docu/files/api/scala/actors/Actor.html#%21%21%28Any%2CPartialFunction%5BAny%2CA%5D%29)

What languages did I miss? From this list, "future" seems to be the
most popular choice, and it doesn't seem to distinguish between the
implicit and explicit kinds.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com