Re: [Python-Dev] PEP 3148 ready for pronouncement

[Paul Moore]

On 24 May 2010 03:58, Cameron Simpson  wrote:
> I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that
> accepts callables in synchronous and asynchronous modes for future
> possibly-concurrent execution, just as the futures module does. I've
> spent a _lot_ of time debugging it.

I pretty much am that user as well (whether or not I am hypothetical,
I'll leave to others to determine...)

I have a set of scripts that needed to do precisely the sort of thing
that the futures module offers. I searched for a fair while for a
suitable offering (this was before futures had been published) and
found nothing suitable. So in the end I implemented my own - and I hit
corner cases, and they needed a lot of work to fix. I now have a
working solution, but it's too tangled in the application logic to be
reusable :-(

If futures had been in the stdlib, I'd have used it like a shot, and
saved myself a lot of wasted time.

> There's a lot to be said for a robust implementation of a well defined
> problem. Brian's module, had it been present and presuming it robust and
> debugged, would have been quite welcome.

Precisely my view.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Georg Brandl]

Am 24.05.2010 01:51, schrieb Greg Ewing:
> Brian Quinlan wrote:
> 
>> The good news in this case is that the same API has been used 
>> successfully in Java and C++ for years so it is unlikely that any major 
>> changes will need to be made.
> 
> That doesn't follow. An API that's appropriate for Java or
> C++ is not necessarily appropriate for Python. Slavishly
> copying an API from another language is often not the best
> approach when designing an API for a Python module.

*cough* unittest *cough*

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] urllib2/urllib incompatibility?

[Vinay Sajip]

> Not top-posting, but gmane seems too finicky in this area.

I encountered what seems like an incompatibility between urllib and urllib2 in
the way they handle file:// URLs, is this a bug? I had a look on the bug tracker
and via Google to see if there were prior reports, but perhaps my search-fu is
deficient. Here's a console session to illustrate:

vi...@eta-karmic:/tmp$ echo Hello, world! >hello.txt
vi...@eta-karmic:/tmp$ cat hello.txt 
Hello, world!
vi...@eta-karmic:/tmp$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib,urllib2
>>> s = 'file:tmp/hello.txt'
>>> f1 = urllib.urlopen(s)
>>> f1.read()
'Hello, world!\n'
>>> f2 = urllib2.urlopen(s)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
'_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1240, in file_open
return self.parent.open(req)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
'_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1287, in ftp_open
raise URLError('ftp error: no host given')
urllib2.URLError: 
>>> 

Anyone seen this before? Should I file an issue on the tracker? If I've missed
something obvious, sorry for the noise, but please do tell!

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Mark Summerfield]

On 2010-05-23, Terry Reedy wrote:
> On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote:
> > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan  wrote:
> >> Rename "executor" =>  "executer"
> >
> > -1 for consistency with Java.
> 
> -10 for 'executer'. As far as I can find out, it is a misspelling of
> 'executor'. If the designers of some other language made a stupid
> mistake, let them correct it instead of us following them over a cliff.

I'd suggested this because it seemed obvious to me, but clearly not.
Compare:
http://www.thefreedictionary.com/executor
http://www.thefreedictionary.com/executer

However, as I mentioned in the first place I didn't expect any change of
this since Java uses the first spelling.

[snip]

-- 
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy
"Advanced Qt Programming" - ISBN 0321635906
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Brian Quinlan]

On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:



On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:


On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:



On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:

Jesse, the designated pronouncer for this PEP, has decided to  
keep discussion open for a few more days.


So fire away!


As you wish!


I retract my request ;-)


May you get what you wish for, may you find what you are seeking :).

The PEP should be consistent in its usage of terminology about  
callables.  It alternately calls them "callables", "functions",  
and "functions or methods".  It would be nice to clean this up and  
be consistent about what can be called where.  I personally like  
"callables".


Did you find the terminology confusing? If not then I propose not  
changing it.


Yes, actually.  Whenever I see references to the multiprocessing  
module, I picture a giant "HERE BE (serialization) DRAGONS" sign.   
When I saw that some things were documented as being "functions", I  
thought that maybe there was intended to be a restriction like the  
"these can only be top-level functions so they're easy for different  
executors to locate and serialize".  I didn't realize that the  
intent was "arbitrary callables" until I carefully re-read the  
document and noticed that the terminology was inconsistent.


ProcessPoolExecutor has the same serialization perils that  
multiprocessing does. My original plan was to link to the  
multiprocessing docs to explain them but I couldn't find them listed.


But changing it in the user docs is probably a good idea. I like  
"callables" too.


Great.  Still, users will inevitably find the PEP and use it as  
documentation too.


The execution context of callable code is not made clear.   
Implicitly, submit() or map() would run the code in threads or  
processes as defined by the executor, but that's not spelled out  
clearly.


Any response to this bit?  Did I miss something in the PEP?


Yes, the execution context is Executor-dependent. The section under  
ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.


More relevant to my own interests, the execution context of the  
callables passed to add_done_callback and remove_done_callback is  
left almost completely to the imagination.  If I'm reading the  
sample implementation correctly, , it looks like in the multiprocessing implementation, the done  
callbacks are invoked in a random local thread.  The fact that  
they are passed the future itself *sort* of implies that this is  
the case, but the multiprocessing module plays fast and loose with  
object identity all over the place, so it would be good to be  
explicit and say that it's *not* a pickled copy of the future  
sitting in some arbitrary process (or even on some arbitrary  
machine).


The callbacks will always be called in a thread other than the main  
thread in the process that created the executor. Is that a strong  
enough contract?


Sure.  Really, almost any contract would work, it just needs to be  
spelled out.  It might be nice to know whether the thread invoking  
the callbacks is a daemon thread or not, but I suppose it's not  
strictly necessary.


Your concerns is that the thread will be killed when the interpreter  
exits? It won't be.


This is really minor, I know, but why does it say "NOTE: This  
method can be used to create adapters from Futures to Twisted  
Deferreds"?  First of all, what's the deal with "NOTE"; it's the  
only "NOTE" in the whole PEP, and it doesn't seem to add  
anything.  This sentence would read exactly the same if that word  
were deleted.  Without more clarity on the required execution  
context of the callbacks, this claim might not actually be true  
anyway; Deferred callbacks can only be invoked in the main reactor  
thread in Twisted.  But even if it is perfectly possible, why  
leave so much of the adapter implementation up to the  
imagination?  If it's important enough to mention, why not have a  
reference to such an adapter in the reference Futures  
implementation, since it *should* be fairly trivial to write?


I'm a bit surprised that this doesn't allow for better  
interoperability with Deferreds given this discussion:






I did not communicate that well.  As implemented, it's quite  
possible to implement a translation layer which turns a Future into  
a Deferred.  What I meant by that comment was, the specification in  
the PEP was to loose to be sure that such a layer would work with  
arbitrary executors.


For what it's worth, the Deferred translator would look like this,  
if you want to include it in the PEP (untested though, you may want  
to run it first):


from twisted.internet.defer import Deferred
from twisted.internet.reactor import callFromThread

def future2deferred(future):
d = Deferred()
def invoke_deferred(

Re: [Python-Dev] urllib2/urllib incompatibility?

[Senthil Kumaran]

On Mon, May 24, 2010 at 08:49:56AM +, Vinay Sajip wrote:
> I encountered what seems like an incompatibility between urllib and urllib2 in
> the way they handle file:// URLs, is this a bug? I had a look on the bug 
> tracker

> >>> s = 'file:tmp/hello.txt'
> >>> f1 = urllib.urlopen(s)

The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt'
And this was fine and consistent.

>>> s = 'file:///tmp/hello.txt'
>>> import urllib2
>>> import urllib
>>> o1 = urllib.urlopen(s)
>>> o2 = urllib2.urlopen(s)


The extra '/' is making it in invalid url in urllib2, I think that be
tracked as bug at least to show a consistent behaviour.  The local
file open methods of urllib2 and urllib are different.

-- 
Senthil

You may my glories and my state dispose,
But not my griefs; still am I king of those.
-- William Shakespeare, "Richard II"
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib2/urllib incompatibility?

[Vinay Sajip]

Senthil Kumaran  gmail.com> writes:

> 
> The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt'
> And this was fine and consistent.
> 
> The extra '/' is making it in invalid url in urllib2, I think that be
> tracked as bug at least to show a consistent behaviour.  The local
> file open methods of urllib2 and urllib are different.
> 

Yes, it seems to be a bug in urllib which permits an invalid URL to pass. I came
across it by accident, I normally use urls like file://localhost/tmp/hello.txt
so I don't trip over the /// hurdles :-)

I'll raise an issue on the tracker.

Regards,

Vinay Sajip


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

[Stephen J. Turnbull]

Cameron Simpson writes:

 > There's a lot to be said for a robust implementation of a well defined
 > problem. Brian's module, had it been present and presuming it robust and
 > debugged, would have been quite welcome.

That, of course, is the consensus view, both in general and with
respect to this particular module.

The difference is over what constitutes sufficient evidence for your
presumption of "robust and debugged" from the point of view of the
users of the stdlib.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] moving issues from argparse tracker to python tracker?

[Antoine Pitrou]

On Sun, 23 May 2010 20:52:19 -0700
Steven Bethard  wrote:
> I have a few feature
> requests, etc. for argparse, and I was planning to just copy them over
> to the Python bug tracker (and close them on the Google code tracker).

Yes, I think this is desireable. You should also maintain argparse
directly in the Python SVN (even if you plan to do separate releases).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __package__ attribute

[Brett Cannon]

On Fri, May 21, 2010 at 14:35, Andrew Svetlov  wrote:
> For me it's the real bug in standard python import machinery.
> I don't see any backward incompatibilities.
> There are very hard to write any import-depended code based on
> decision: was module imported in absolute or relative way.
>
> If it's a bug I can create issue in python bugtracker and provide a
> patch (for Python 3.2 and for 2.7 if later is required).

It's definitely not a bug. The definition of __package__ is such that
not bothering to set it is still valid (see PEP 366 for the exact
details of the attribute). It just happens that importlib puts in the
extra effort to add/set the attribute as much as possible. Now if you
want to write a patch for import.c to add the attribute properly then
that could get reviewed and added as a feature request, but as it
stands the current semantics are correct.

-Brett

>
> On Fri, May 21, 2010 at 10:18 PM, Andrew Svetlov
>  wrote:
>> What are expected values for module.__package__?
>> I see two different behaviors: importlib and standard python import.
>> importlib processes __package__ pretty simple and clean:
>> - for toplevel modules __package__ is empty string
>> - for packages it's package name
>> - for modules inside packages it's again package name
>>
>> Standard import follows another strategy:
>> - it tries to setup __package__ only in case of relative import (see
>> first 'if' in import.c:get_parent)
>> - otherwise __package__ is untouched and None by default.
>>
>> For me importlib's way is better.
>> I don't see any PEP specifying __package__ behavior.
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com