Re: [Python-Dev] PEP 3148 ready for pronouncement

[geremy condra]

On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan  wrote:



> Finally, why isn't this just a module on PyPI?  It doesn't seem like there's
> any particular benefit to making this a stdlib module and going through the
> whole PEP process - except maybe to prompt feedback like this :).
>
> We've already had this discussion before. Could you explain why this module
> should *not* be in the stdlib e.g. does it have significantly less utility
> than other modules in stdlib? Is it significantly higher risk? etc?

Inclusion in the stdlib is the exception, not the rule, and every
exception should be issued for a good reason. I'd like to know
what that reason is in this case, if only to get a clearer
understanding of why the PEP was accepted.

> Issues like the ones I'm bringing up could be fixed pretty straightforwardly
> if it were just a matter of filing a bug on a small package, but fixing a
> stdlib module is a major undertaking.
>
> True but I don't think that is a convincing argument. A subset of the
> functionality provided by this module is already available in Java and C++
> and (at least in Java) it is used extensively and without too much trouble.
> If there are implementation bugs then we can fix them just like we would
> with any other module.

Guido made exactly the opposite argument during his keynote at PyCon.
It seemed fairly reasonable at the time- why do you think it doesn't apply
here?

Geremy Condra
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On May 23, 2010, at 7:15 PM, geremy condra wrote:

On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan   
wrote:




Finally, why isn't this just a module on PyPI?  It doesn't seem  
like there's
any particular benefit to making this a stdlib module and going  
through the

whole PEP process - except maybe to prompt feedback like this :).

We've already had this discussion before. Could you explain why  
this module
should *not* be in the stdlib e.g. does it have significantly less  
utility

than other modules in stdlib? Is it significantly higher risk? etc?


Inclusion in the stdlib is the exception, not the rule, and every
exception should be issued for a good reason. I'd like to know
what that reason is in this case,


This package eliminates the need to construct the boilerplate present  
in many Python applications i.e. a thread or process pool, a work  
queue and result queue.  It also makes it easy to take an existing  
Python application that executes (e.g. IO operations) in sequence and  
execute them in parallel. It package provides common idioms for two  
existing modules i.e. multiprocessing offers map functionality while  
threading doesn't. Those idioms are well understood and already  
present in Java and C++.



if only to get a clearer
understanding of why the PEP was accepted.


It hasn't been accepted.

Issues like the ones I'm bringing up could be fixed pretty  
straightforwardly
if it were just a matter of filing a bug on a small package, but  
fixing a

stdlib module is a major undertaking.

True but I don't think that is a convincing argument. A subset of the
functionality provided by this module is already available in Java  
and C++
and (at least in Java) it is used extensively and without too much  
trouble.
If there are implementation bugs then we can fix them just like we  
would

with any other module.


Guido made exactly the opposite argument during his keynote at PyCon.
It seemed fairly reasonable at the time- why do you think it doesn't  
apply

here?



Could you be a little more specific about Guido's argument at PyCon?

Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Lennart Regebro]

On Sun, May 23, 2010 at 11:39, Brian Quinlan  wrote:
> This package eliminates the need to construct the boilerplate present in
> many Python applications i.e. a thread or process pool, a work queue and
> result queue.  It also makes it easy to take an existing Python application
> that executes (e.g. IO operations) in sequence and execute them in parallel.
> It package provides common idioms for two existing modules i.e.
> multiprocessing offers map functionality while threading doesn't. Those
> idioms are well understood and already present in Java and C++.

It can do that as a separate package as well. And not only that, it
could then be available on PyPI for earlier versions of Python as
well, making it much more likely to gain widespread acceptance.

> Could you be a little more specific about Guido's argument at PyCon?

A module in stdlib has to be "dead". After it's included in the stdlib
it can not go through any major changes since that would mean loss of
backwards incompatibility. Also, you can't fix bugs except by
releasing new versions of Python. Therefore the API must be completely
stable, and the product virtually bugfree before it should be in
stdlib. The best way of ensuring that is to release it as a separate
module on PyPI, and let it stabilize for a couple of years.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On May 23, 2010, at 7:54 PM, Lennart Regebro wrote:

On Sun, May 23, 2010 at 11:39, Brian Quinlan   
wrote:
This package eliminates the need to construct the boilerplate  
present in
many Python applications i.e. a thread or process pool, a work  
queue and
result queue.  It also makes it easy to take an existing Python  
application
that executes (e.g. IO operations) in sequence and execute them in  
parallel.

It package provides common idioms for two existing modules i.e.
multiprocessing offers map functionality while threading doesn't.  
Those

idioms are well understood and already present in Java and C++.


It can do that as a separate package as well.


You could make the same argument about any module in the stdlib.


And not only that, it
could then be available on PyPI for earlier versions of Python as
well, making it much more likely to gain widespread acceptance.


I doubt it. Simple modules are unlikely to develop a following because  
it is too easy to partially replicate their functionality. urlparse  
and os.path are very useful modules but I doubt that they would have  
been successful on PyPI.



Could you be a little more specific about Guido's argument at PyCon?


A module in stdlib has to be "dead". After it's included in the stdlib
it can not go through any major changes since that would mean loss of
backwards incompatibility.


The good news in this case is that the same API has been used  
successfully in Java and C++ for years so it is unlikely that any  
major changes will need to be made.



Also, you can't fix bugs except by
releasing new versions of Python. Therefore the API must be completely
stable, and the product virtually bugfree before it should be in
stdlib. The best way of ensuring that is to release it as a separate
module on PyPI, and let it stabilize for a couple of years.



Yeah but that model isn't likely to work with this package.

Cheers,
Brian___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Robert Collins]

On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan  wrote:

>> Also, you can't fix bugs except by
>> releasing new versions of Python. Therefore the API must be completely
>> stable, and the product virtually bugfree before it should be in
>> stdlib. The best way of ensuring that is to release it as a separate
>> module on PyPI, and let it stabilize for a couple of years.
>
> Yeah but that model isn't likely to work with this package.
> Cheers,
> Brian

Forgive my ignorance, but why do you say that that model won't work
with this package?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Dirkjan Ochtman]

On Sun, May 23, 2010 at 12:15, Brian Quinlan  wrote:
> I doubt it. Simple modules are unlikely to develop a following because it is
> too easy to partially replicate their functionality. urlparse and os.path
> are very useful modules but I doubt that they would have been successful on
> PyPI.

simplejson was also fairly simple, but still developed a following.

> The good news in this case is that the same API has been used successfully
> in Java and C++ for years so it is unlikely that any major changes will need
> to be made.

I would agree that having prior versions in other languages should
make the API more stable, but I wouldn't agree that it doesn't need
changes (and even minor changes can be a PITA in the stdlib).

>> Also, you can't fix bugs except by
>> releasing new versions of Python. Therefore the API must be completely
>> stable, and the product virtually bugfree before it should be in
>> stdlib. The best way of ensuring that is to release it as a separate
>> module on PyPI, and let it stabilize for a couple of years.
>
> Yeah but that model isn't likely to work with this package.

Okay, I'll bite: why is your package different?

In general, this reminds me of the ipaddr discussions. I read through
the thread from March real quick to see if there was reasoning there
why this package should be an exception from the "normal" standards
track (that is, ripen on PyPI, then moving it in the stdlib when it's
mature -- where "mature" is another word for dead, really). But then
this is just another instance of the fat-stdlib vs lean-stdlib
discussion, I guess, so we can go on at length.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On May 23, 2010, at 8:43 PM, Robert Collins wrote:

On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan   
wrote:



Also, you can't fix bugs except by
releasing new versions of Python. Therefore the API must be  
completely

stable, and the product virtually bugfree before it should be in
stdlib. The best way of ensuring that is to release it as a separate
module on PyPI, and let it stabilize for a couple of years.


Yeah but that model isn't likely to work with this package.
Cheers,
Brian


Forgive my ignorance, but why do you say that that model won't work
with this package?


As I said in my last message:

"""Simple modules are unlikely to develop a following because it is  
too easy to partially replicate their functionality. urlparse and  
os.path are very useful modules but I doubt that they would have been  
successful on PyPI."""


Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Dead modules

[Antoine Pitrou]

On Sun, 23 May 2010 12:43:57 +0200
Dirkjan Ochtman  wrote:
> 
> In general, this reminds me of the ipaddr discussions. I read through
> the thread from March real quick to see if there was reasoning there
> why this package should be an exception from the "normal" standards
> track (that is, ripen on PyPI, then moving it in the stdlib when it's
> mature -- where "mature" is another word for dead, really).

I disagree that a stdlib module is a dead module. It is perfectly
possible to augment the API with new functionality without breaking
compatibility. You can also deprecate old APIs if you want.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On May 23, 2010, at 8:43 PM, Dirkjan Ochtman wrote:
On Sun, May 23, 2010 at 12:15, Brian Quinlan   
wrote:
I doubt it. Simple modules are unlikely to develop a following  
because it is
too easy to partially replicate their functionality. urlparse and  
os.path
are very useful modules but I doubt that they would have been  
successful on

PyPI.


simplejson was also fairly simple, but still developed a following.


The API is simple but writing a JSON parser is hard enough that people  
will check to see if someone has already done the work for them  
(especially since JSON is fairly topical).


If you are familiar with threads then writing a "good enough" solution  
without futures probably won't take you very long. Also, unless you  
are familiar with another futures implementation, you aren't likely to  
know where to look.


The good news in this case is that the same API has been used  
successfully
in Java and C++ for years so it is unlikely that any major changes  
will need

to be made.


I would agree that having prior versions in other languages should
make the API more stable, but I wouldn't agree that it doesn't need
changes (and even minor changes can be a PITA in the stdlib).


Some changes are hard (i.e. changing the semantics of existing method)  
but some are pretty easy (i.e. adding new methods).


Cheers,
Brian



Also, you can't fix bugs except by
releasing new versions of Python. Therefore the API must be  
completely

stable, and the product virtually bugfree before it should be in
stdlib. The best way of ensuring that is to release it as a separate
module on PyPI, and let it stabilize for a couple of years.


Yeah but that model isn't likely to work with this package.


Okay, I'll bite: why is your package different?

In general, this reminds me of the ipaddr discussions. I read through
the thread from March real quick to see if there was reasoning there
why this package should be an exception from the "normal" standards
track (that is, ripen on PyPI, then moving it in the stdlib when it's
mature -- where "mature" is another word for dead, really). But then
this is just another instance of the fat-stdlib vs lean-stdlib
discussion, I guess, so we can go on at length.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dead modules

[Dirkjan Ochtman]

On Sun, May 23, 2010 at 12:51, Antoine Pitrou  wrote:
> I disagree that a stdlib module is a dead module. It is perfectly
> possible to augment the API with new functionality without breaking
> compatibility. You can also deprecate old APIs if you want.

Right, it wasn't intended as that harsh... but it does come with a
rather impressive set of constraints in terms of what you can do with
the API.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Andrew Svetlov]

Is there any reason to have Future .cancelled, .done, .running as methods?
>From my perspective they are really readonly properties.

BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
form: 'canceled'.

On Sun, May 23, 2010 at 2:47 PM, Brian Quinlan  wrote:
>
> On May 23, 2010, at 8:43 PM, Robert Collins wrote:
>
>> On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan 
>> wrote:
>>
 Also, you can't fix bugs except by
 releasing new versions of Python. Therefore the API must be completely
 stable, and the product virtually bugfree before it should be in
 stdlib. The best way of ensuring that is to release it as a separate
 module on PyPI, and let it stabilize for a couple of years.
>>>
>>> Yeah but that model isn't likely to work with this package.
>>> Cheers,
>>> Brian
>>
>> Forgive my ignorance, but why do you say that that model won't work
>> with this package?
>
> As I said in my last message:
>
> """Simple modules are unlikely to develop a following because it is too easy
> to partially replicate their functionality. urlparse and os.path are very
> useful modules but I doubt that they would have been successful on PyPI."""
>
> Cheers,
> Brian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Lennart Regebro]

On Sun, May 23, 2010 at 12:15, Brian Quinlan  wrote:
> You could make the same argument about any module in the stdlib.

Yeah, and that's exactly what I did.

> I doubt it. Simple modules are unlikely to develop a following because it is
> too easy to partially replicate their functionality. urlparse and os.path
> are very useful modules but I doubt that they would have been successful on
> PyPI.

Are you saying your proposed module is so simple that anyone can
easily replicate it with just a couple of lines of code?

> The good news in this case is that the same API has been used successfully
> in Java and C++ for years so it is unlikely that any major changes will need
> to be made.

Good. Then the time it takes to "mature" on PyPI would be very short.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jon Ribbens]

On Sun, May 23, 2010 at 03:16:27PM +0400, Andrew Svetlov wrote:
> Is there any reason to have Future .cancelled, .done, .running as methods?
> >From my perspective they are really readonly properties.
> 
> BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
> form: 'canceled'.

In English, only "cancelled" is correct. In American, either is correct.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On 23 May 2010, at 21:17, Lennart Regebro wrote:

On Sun, May 23, 2010 at 12:15, Brian Quinlan   
wrote:

You could make the same argument about any module in the stdlib.


Yeah, and that's exactly what I did.

I doubt it. Simple modules are unlikely to develop a following  
because it is
too easy to partially replicate their functionality. urlparse and  
os.path
are very useful modules but I doubt that they would have been  
successful on

PyPI.


Are you saying your proposed module is so simple that anyone can
easily replicate it with just a couple of lines of code?


Parts of it, yes. Just like I can replace most operations in os.path  
and urlparse with a few lines of code.


The good news in this case is that the same API has been used  
successfully
in Java and C++ for years so it is unlikely that any major changes  
will need

to be made.


Good. Then the time it takes to "mature" on PyPI would be very short.


How would you define "very short"? I've had the project on PyPI for  
about a year now:

http://pypi.python.org/pypi/futures3

Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Steve Holden]

Brian Quinlan wrote:
> 
> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
> 
[...]
>> One minor suggestion on the "internal future methods" bit - something
>> I wish we'd done with Deferreds was to put 'callback()' and
>> 'addCallbacks()' on separate objects, so that it was very explicit
>> whether you were on the emitting side of a Deferred or the consuming
>> side.  That seems to be the case with these internal methods - they
>> are not so much "internal" as they are for the producer of the Future
>> (whether a unit test or executor) so you might want to put them on a
>> different object that it's easy for the thing creating a Future() to
>> get at but hard for any subsequent application code to fiddle with by
>> accident.  Off the top of my head, I suggest naming it "Invoker()".  A
>> good way to do this would be to have an Invoker class which can't be
>> instantiated (raises an exception from __init__ or somesuch), then a
>> Future.create() method which returns an Invoker, which itself has a
>> '.future' attribute.
>>
>> Finally, why isn't this just a module on PyPI?  It doesn't seem like
>> there's any particular benefit to making this a stdlib module and
>> going through the whole PEP process - except maybe to prompt feedback
>> like this :).
> 
> We've already had this discussion before. Could you explain why this
> module should *not* be in the stdlib e.g. does it have significantly
> less utility than other modules in stdlib? Is it significantly higher
> risk? etc?
> 
Given that its author was ready to go for pronouncement and is still
responding to pretty serious philosophical questions about the API I'd
say that it was at least worth talking about. The thing that's needed
(isn't it?) of stdlib modules is API stability.

> 
>> Issues like the ones I'm bringing up could be fixed pretty
>> straightforwardly if it were just a matter of filing a bug on a small
>> package, but fixing a stdlib module is a major undertaking.
> 
> True but I don't think that is a convincing argument. A subset of the
> functionality provided by this module is already available in Java and
> C++ and (at least in Java) it is used extensively and without too much
> trouble. If there are implementation bugs then we can fix them just like
> we would with any other module.
> 
I don't see the availability of this functionality in those languages as
any kind of reason why this needs to go into the stdlib now. Is there
some desperate rush to get it in? If it were used extensively from PyPi
*that* would be a recommendation ...

regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
See Python Video!   http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS:http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
 Ian Dury, 1942-2000

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Lennart Regebro]

On Sun, May 23, 2010 at 13:29, Brian Quinlan  wrote:
> Parts of it, yes. Just like I can replace most operations in os.path and
> urlparse with a few lines of code.

Yeah, but "parts of" is not the question. I've read the PEP, and I do
*not* know how to implement it. That means it's not a trivial module,
so that argument doesn't hold up here, even if we accept it as valid
(which I actually don't). I don't think any module in the stdlib is
entirely trivial. Yes, even parsing an URL is non-trivial, as shown by
the fact that the urlparse module apparently has a bug in it for urls
like svn+ssh://foo.bar/frotz. ;-)

Also, even trivial modules can be useful if you use them a lot.

> How would you define "very short"?

That's not up to me to decide.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Jesse Noller]

On Sun, May 23, 2010 at 7:52 AM, Steve Holden  wrote:
...snip...

>>
>>> Issues like the ones I'm bringing up could be fixed pretty
>>> straightforwardly if it were just a matter of filing a bug on a small
>>> package, but fixing a stdlib module is a major undertaking.
>>
>> True but I don't think that is a convincing argument. A subset of the
>> functionality provided by this module is already available in Java and
>> C++ and (at least in Java) it is used extensively and without too much
>> trouble. If there are implementation bugs then we can fix them just like
>> we would with any other module.
>>
> I don't see the availability of this functionality in those languages as
> any kind of reason why this needs to go into the stdlib now. Is there
> some desperate rush to get it in? If it were used extensively from PyPi
> *that* would be a recommendation ...
>

Not picking Steve's particular comments out - but Brian cites the
previous discussions in the PEP itself:

http://www.python.org/dev/peps/pep-3148/

All of you questioning "Why should this be in the standard library"
should go read those old threads, where that question was answered
numerous times. Now I suddenly regret leaving the floodgates open, as
we're rapidly rehashing discussions from months ago.

For this same mailing list only a few months ago (brian, I think this
link should be added to the PEP, I didn't see it):
http://mail.python.org/pipermail/python-dev/2010-March/098169.html

Specifically:
http://mail.python.org/pipermail/python-dev/2010-March/098173.html

Quote:

"Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package - but Brian's one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it's good
sugar, and is used by a fair number of people.

Let me also state - "my" vision of where futures would live would be
in a concurrent package - for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.
"

Brian has already agreed to name spacing it to "concurrent.futures" -
this means it will be a small part to a much larger concurrent.*
implementation ala Java.

So, in short - given we've already hashed the reasoning out.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Antoine Pitrou]

On Sun, 23 May 2010 08:34:22 -0400
Jesse Noller  wrote:
> 
> Brian has already agreed to name spacing it to "concurrent.futures" -
> this means it will be a small part to a much larger concurrent.*
> implementation ala Java.

What I would question here is what other things will be part
of the "concurrent" package, and who will implement them. Are there
plans for that? (or even tracker issues open?)

Apart from that, it seems to me that the only serious issues
blocking PEP approval are Glyph's interesting remarks.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Steve Holden]

Jesse Noller wrote:
> On Sun, May 23, 2010 at 7:52 AM, Steve Holden  wrote:
> ...snip...
> 
 Issues like the ones I'm bringing up could be fixed pretty
 straightforwardly if it were just a matter of filing a bug on a small
 package, but fixing a stdlib module is a major undertaking.
>>> True but I don't think that is a convincing argument. A subset of the
>>> functionality provided by this module is already available in Java and
>>> C++ and (at least in Java) it is used extensively and without too much
>>> trouble. If there are implementation bugs then we can fix them just like
>>> we would with any other module.
>>>
>> I don't see the availability of this functionality in those languages as
>> any kind of reason why this needs to go into the stdlib now. Is there
>> some desperate rush to get it in? If it were used extensively from PyPi
>> *that* would be a recommendation ...
>>
> 
> Not picking Steve's particular comments out - but Brian cites the
> previous discussions in the PEP itself:
> 
> http://www.python.org/dev/peps/pep-3148/
> 
> All of you questioning "Why should this be in the standard library"
> should go read those old threads, where that question was answered
> numerous times. Now I suddenly regret leaving the floodgates open, as
> we're rapidly rehashing discussions from months ago.
> 
Yes, it might have been better to call for participation from those who
had contributed to the original discussion, and therefore knew what they
were talking about. No flood from me, though, all my questions have been
answered.

regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
See Python Video!   http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS:http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
 Ian Dury, 1942-2000
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Terry Reedy]

On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote:

On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan  wrote:

Rename "executor" =>  "executer"


-1 for consistency with Java.


-10 for 'executer'. As far as I can find out, it is a misspelling of 
'executor'. If the designers of some other language made a stupid 
mistake, let them correct it instead of us following them over a cliff.


Unlike this one, the other name suggestions look at least plausible and 
worth a couple of minutes of consideration.


As for the module itself, part of the justification to me for accepting 
it would be if it is part of a larger plan, even if currently vague, to 
refactor and improve Python's support for concurrent execution, as 
implied by the name 'concurrent.futures'. If Jesse accepts it, I would 
take it as some kind of commitment to help with at least one other 
concurrent.x module so this one is not an orphan. While concurrent 
execution does not *currently* affect me, I am convinced that better 
support will be important for Python's future.


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Glyph Lefkowitz]

On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:

> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
> 
>> 
>> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:
>> 
>>> Jesse, the designated pronouncer for this PEP, has decided to keep 
>>> discussion open for a few more days.
>>> 
>>> So fire away!
>> 
>> As you wish!
> 
> I retract my request ;-)

May you get what you wish for, may you find what you are seeking :).

>> The PEP should be consistent in its usage of terminology about callables.  
>> It alternately calls them "callables", "functions", and "functions or 
>> methods".  It would be nice to clean this up and be consistent about what 
>> can be called where.  I personally like "callables".
> 
> Did you find the terminology confusing? If not then I propose not changing it.

Yes, actually.  Whenever I see references to the multiprocessing module, I 
picture a giant "HERE BE (serialization) DRAGONS" sign.  When I saw that some 
things were documented as being "functions", I thought that maybe there was 
intended to be a restriction like the "these can only be top-level functions so 
they're easy for different executors to locate and serialize".  I didn't 
realize that the intent was "arbitrary callables" until I carefully re-read the 
document and noticed that the terminology was inconsistent.

> But changing it in the user docs is probably a good idea. I like "callables" 
> too.

Great.  Still, users will inevitably find the PEP and use it as documentation 
too.

>> The execution context of callable code is not made clear.  Implicitly, 
>> submit() or map() would run the code in threads or processes as defined by 
>> the executor, but that's not spelled out clearly.

Any response to this bit?  Did I miss something in the PEP?

>> More relevant to my own interests, the execution context of the callables 
>> passed to add_done_callback and remove_done_callback is left almost 
>> completely to the imagination.  If I'm reading the sample implementation 
>> correctly, 
>> ,
>>  it looks like in the multiprocessing implementation, the done callbacks are 
>> invoked in a random local thread.  The fact that they are passed the future 
>> itself *sort* of implies that this is the case, but the multiprocessing 
>> module plays fast and loose with object identity all over the place, so it 
>> would be good to be explicit and say that it's *not* a pickled copy of the 
>> future sitting in some arbitrary process (or even on some arbitrary machine).
> 
> The callbacks will always be called in a thread other than the main thread in 
> the process that created the executor. Is that a strong enough contract?

Sure.  Really, almost any contract would work, it just needs to be spelled out. 
 It might be nice to know whether the thread invoking the callbacks is a daemon 
thread or not, but I suppose it's not strictly necessary.

>> This is really minor, I know, but why does it say "NOTE: This method can be 
>> used to create adapters from Futures to Twisted Deferreds"?  First of all, 
>> what's the deal with "NOTE"; it's the only "NOTE" in the whole PEP, and it 
>> doesn't seem to add anything.  This sentence would read exactly the same if 
>> that word were deleted.  Without more clarity on the required execution 
>> context of the callbacks, this claim might not actually be true anyway; 
>> Deferred callbacks can only be invoked in the main reactor thread in 
>> Twisted.  But even if it is perfectly possible, why leave so much of the 
>> adapter implementation up to the imagination?  If it's important enough to 
>> mention, why not have a reference to such an adapter in the reference 
>> Futures implementation, since it *should* be fairly trivial to write?
> 
> I'm a bit surprised that this doesn't allow for better interoperability with 
> Deferreds given this discussion:

> 

I did not communicate that well.  As implemented, it's quite possible to 
implement a translation layer which turns a Future into a Deferred.  What I 
meant by that comment was, the specification in the PEP was to loose to be sure 
that such a layer would work with arbitrary executors.

For what it's worth, the Deferred translator would look like this, if you want 
to include it in the PEP (untested though, you may want to run it first):

from twisted.internet.defer import Deferred
from twisted.internet.reactor import callFromThread

def future2deferred(future):
d = Deferred()
def invoke_deferred():
try:
result = future.result()
except:
d.errback()
else:
d.callback(result)
def done_callback(same_future):
callFromThread(invoke_deferred)
future.add_done_callback(done_callback)
return d

This does beg the question of what the traceback will look like in that except: 
block though.  I guess the m

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Greg Ewing]

Glyph Lefkowitz wrote:

Finally, why isn't this just a module on PyPI?  It doesn't seem like 
there's any particular benefit to making this a stdlib module and going 
through the whole PEP process


I'm inclined to agree. This needs to be field-tested before
being considered for stdlib inclusion.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Greg Ewing]

Brian Quinlan wrote:

The good news in this case is that the same API has been used 
successfully in Java and C++ for years so it is unlikely that any major 
changes will need to be made.


That doesn't follow. An API that's appropriate for Java or
C++ is not necessarily appropriate for Python. Slavishly
copying an API from another language is often not the best
approach when designing an API for a Python module.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Greg Ewing]

Brian Quinlan wrote:

Simple modules are unlikely to develop a following because it is  too 
easy to partially replicate their functionality.


I don't think it needs a particularly large following.
What it does need is at least a few people using it in
some real projects.

No matter how much discussion there is and how much
apparent agreement is reached, it's no substitute for
practical experience. Often API design mistakes are only
found when trying to use the library for real.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Greg Ewing]

Andrew Svetlov wrote:


BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
form: 'canceled'.


I think this is an English vs. American thing. Double 'l'
looks right to me, but then I was brought up as a loyal
subject of the antipodean branch of the British Empire. :-)

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Stephen J. Turnbull]

Brian Quinlan writes:

 > If you are familiar with threads then writing a "good enough" solution  
 > without futures probably won't take you very long. Also, unless you  
 > are familiar with another futures implementation, you aren't likely to  
 > know where to look.

That looks like an argument *against* your module, to me.  Why would
people look for it in the stdlib if they're not looking for it at all,
and specifically because anybody who would know enough to look for
"something like" it is also able to devise a good-enough solution?
You're describing a solution in search of a user, not a user in search
of a solution, and it would appear to violate "not every three-line
function" as well as TOOWTDI.

I personally plan to defer to the people who know and use such
constructs (specifically Glyph and Jesse), and who seem to be in favor
(at least +0) of stabilizing an API for this in the stdlib.  But you
may want to rethink your sales pitch if you want to avoid giving ammo
to the opposition.  It sounds like you agree with them, except on the
vote you cast.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Cameron Simpson]

On 24May2010 10:47, Stephen J. Turnbull  wrote:
| Brian Quinlan writes:
|  > If you are familiar with threads then writing a "good enough" solution  
|  > without futures probably won't take you very long. Also, unless you  
|  > are familiar with another futures implementation, you aren't likely to  
|  > know where to look.
| 
| That looks like an argument *against* your module, to me.  Why would
| people look for it in the stdlib if they're not looking for it at all,
| and specifically because anybody who would know enough to look for
| "something like" it is also able to devise a good-enough solution?
| You're describing a solution in search of a user, not a user in search
| of a solution, and it would appear to violate "not every three-line
| function" as well as TOOWTDI.

This might be a terminology problem. I think, above, Brian means "good
enough" to mean "looks ok at first cut but doesn't handle the corner
cases". Which usually means obscure breakage later.

I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that
accepts callables in synchronous and asynchronous modes for future
possibly-concurrent execution, just as the futures module does. I've
spent a _lot_ of time debugging it.

There's a lot to be said for a robust implementation of a well defined
problem. Brian's module, had it been present and presuming it robust and
debugged, would have been quite welcome.

Cheers,
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

I am a Bear of Very Little Brain and long words Bother Me.
- Winnie-the-Pooh
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] moving issues from argparse tracker to python tracker?

[Steven Bethard]

Before I go and add about 30 open issues to the Python tracker, I
figured I should ask. What's the normal process for the bug trackers
of modules that move to the standard library? I have a few feature
requests, etc. for argparse, and I was planning to just copy them over
to the Python bug tracker (and close them on the Google code tracker).
Is this what people normally do? (It should be easy enough to do - I
just don't want to mess up the tracker if this is usually done some
other way.)

Thanks,

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] moving issues from argparse tracker to python tracker?

[Senthil Kumaran]

On Sun, May 23, 2010 at 08:52:19PM -0700, Steven Bethard wrote:
> requests, etc. for argparse, and I was planning to just copy them over
> to the Python bug tracker (and close them on the Google code tracker).

I think, this is a good idea. +1 from me. There is only one module,
bsddb, which is maintained outside due to release constraints and
other issues, but other than anything part of stdlib, better be
tracked through python tracker.

An additional advantage of moving to python tracker is, people who are
subscribed to python-bugs will be notified and can pitch in their
ideas/efforts too.

-- 
Senthil

OK, enough hype.
-- Larry Wall in the perl man page
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com