Re: [Python-Dev] Adding resume (206) support to urllib(2)

2006-12-13 Thread Martin v. Löwis 
Oleg Broytmann schrieb:
>IMO you better don't because urllib2 provides not only an abstraction,
> but a lot of services (authenticated proxies, cached FTP files)...

If you are using http ranges, cached FTP files won't do any good.
As for authenticated proxies: I think they ought to be implemented
in httplib as well.

If everybody wants to become urllib just a better library to access
http servers, I probably can't do much about it, though.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding resume (206) support to urllib(2)

2006-12-13 Thread Oleg Broytmann 
On Wed, Dec 13, 2006 at 09:05:49AM +0100, "Martin v. L?wis" wrote:
> As for authenticated proxies: I think they ought to be implemented
> in httplib as well.

   Agree.

> If everybody wants to become urllib just a better library to access
> http servers, I probably can't do much about it, though.

   HTTP is one of the most widely known and used protocol. Would you better
have big httplib and small abstract urllib? so abstract it doesn't allow a
user to change protocol-specific handling?

Oleg.
-- 
 Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED]
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?!

2006-12-13 Thread Terry Jones 
> "Guido" == Guido van Rossum <[EMAIL PROTECTED]> writes:
Guido> And I just found out (after everyone else probably :-) that YouTube
Guido> is almost entirely written in Python. (And now I can rub shoulders
Guido> with the developers since they're all Googlers now... :-)

Are any other details available? I'm mainly curious to know if they were
using Twisted.

Terry
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Creating dicts from dict subclasses

2006-12-13 Thread Walter Dörwald 
I tried to reimplement weakref.WeakValueDictionary as a subclass of
dict. The test passes except for one problem: To compare results
test_weakref.py converts a weakdict to a real dict via dict(weakdict).
This no longer works because PyDict_Merge() does a PyDict_Check() on the
argument and then ignores all overwritten methods. (The old version
worked because UserDict.UserDict was used).

The simplest solution is to replace the PyDict_Check() call with
PyDict_CheckExact(), but this might slow things down too much, because
the fallback code basically does:

   for key in iter(arg.keys()):
  self[key] = arg.__getitem__(key)

Why can't we use:

   for key in iter(arg):
  self[key] = arg.__getitem__(key)

instead?

Servus,
   Walter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?!

2006-12-13 Thread Guido van Rossum 
No they aren't using Twisted.

On 12/12/06, Terry Jones <[EMAIL PROTECTED]> wrote:
> > "Guido" == Guido van Rossum <[EMAIL PROTECTED]> writes:
> Guido> And I just found out (after everyone else probably :-) that YouTube
> Guido> is almost entirely written in Python. (And now I can rub shoulders
> Guido> with the developers since they're all Googlers now... :-)
>
> Are any other details available? I'm mainly curious to know if they were
> using Twisted.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Creating dicts from dict subclasses

2006-12-13 Thread Guido van Rossum 
On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> I tried to reimplement weakref.WeakValueDictionary as a subclass of
> dict. The test passes except for one problem: To compare results
> test_weakref.py converts a weakdict to a real dict via dict(weakdict).
> This no longer works because PyDict_Merge() does a PyDict_Check() on the
> argument and then ignores all overwritten methods. (The old version
> worked because UserDict.UserDict was used).
>
> The simplest solution is to replace the PyDict_Check() call with
> PyDict_CheckExact(), but this might slow things down too much, because
> the fallback code basically does:
>
>for key in iter(arg.keys()):
>   self[key] = arg.__getitem__(key)
>
> Why can't we use:
>
>for key in iter(arg):
>   self[key] = arg.__getitem__(key)
>
> instead?

The only reason I can think of is backwards compatibility: not all
"mappings" created pre-2.2 would support iteration. Maybe you could
check for a tp_iter slot and if non-NULL use the latter otherwise use
the original fallback?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Open CPython VM for type information

2006-12-13 Thread Brett Cannon 

On 12/12/06, Kay Schluehr <[EMAIL PROTECTED]> wrote:


Dear Python developers,

while the JVM is opened to support dynamically typed languages [1] I
wonder if the CPyVM could not show more openness to statically typed
languages? Hereby I don't think that much about arbitrary languages for
the CPyVM but sublanguages like RPython which are "static enough" to be
compilable into extension modules [2].

However creating C extension modules from Python code is not really the
standard way I do think Python source should be handled. Instead I think
about a more direct VM support like the JVM does, that provides opcodes
like iload, fstore etc. Besides this I don't expect one canonical
approach for these type information to be feeded into the CPyVM. There
are type inferencer like rtyper for PyPy, Psycos continous bytecode
monitoring and also other approaches are possible that use type
feedback. Finally optional type declarations are also an issue; however
I would like to see language design discussions being separated ( ->
Python 3000 ) and focus on general APIs for compiler extensions / plugs
that deal with type information ( the architecture ) and VM
implementation issues.

Thanks, Kay

[1] http://jcp.org/en/jsr/detail?id=292
[2] http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html



The Python VM is not designed to be general-purpose, period.  Nor does
anyone on python-dev, to my knowledge, have any interest in putting the time
and energy into making it general.  I am quite happy with letting the Parrot
folk handle this goal.  This means that the maintenance cost of adding more
opcodes is not worth it.

But a bigger issue with me is the performance hit.  Every opcode will make
the eval loop larger.  And it is known that the eval loop is rather finicky
in terms of size and performance.  While no hard testing has been done, it
has been assumed we have accidentally hit the sweet spot in terms of CPU
cache size.

An important thing to keep in mind when comparing Python's VM to Java's is
that Java works (somewhat) for dynamic languages because people are
motivated enough to shoehorn languages into it by just assuming everything
is just an object and tossing out the type checking inherent to Java.  But
Java has not, to my knowledge, grown any great features yet to make dynamic
languages work better.  I know about JSR 292, but that is not in yet and
adding just one opcode is not that expensive to the JVM.  But compare the
cost of adding a slew of typed opcodes to Python compared to the JVM adding
just one is not really equal.

In other words, I wouldn't count on this ever happening.  =)  The best you
could do is try to propose some direct mechanism for adding support to the
eval loop to dispatch to some function for unrecognized opcodes, but I
wouldn't count on that being accepted either.

-Brett
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding resume (206) support to urllib(2)

2006-12-13 Thread Martin v. Löwis 
Oleg Broytmann schrieb:
>HTTP is one of the most widely known and used protocol. Would you better
> have big httplib and small abstract urllib? so abstract it doesn't allow a
> user to change protocol-specific handling?

Personally, I think very elaborate support for HTTP in httplib, and very
few generalizations and abstractions in urllib* would be the "right"
way to do it, IMO. For example, there might be the notion of an
"http session" object where a single application request can resolve
to multiple http requests (with redirection, authentication negotiation,
cookies, 100 continue, implicit headers, etc).

For compatibility, urllib* can't drop features, and we'd need
contributors who contribute such a refactoring, but IMO that would
be the right way. If applications use urllib *only* for http, and
*only* because it has these multi-request, implicit headers
features, something is wrong with the abstractions.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding resume (206) support to urllib(2)

2006-12-13 Thread Oleg Broytmann 
On Wed, Dec 13, 2006 at 10:00:48PM +0100, "Martin v. L?wis" wrote:
> Personally, I think very elaborate support for HTTP in httplib, and very
> few generalizations and abstractions in urllib* would be the "right"
> way to do it, IMO. For example, there might be the notion of an
> "http session" object where a single application request can resolve
> to multiple http requests (with redirection, authentication negotiation,
> cookies, 100 continue, implicit headers, etc).

   I see.

> For compatibility, urllib* can't drop features

   Leave it for py3k, then.

> and we'd need
> contributors who contribute such a refactoring

   That's the hardest part.

> If applications use urllib *only* for http, and
> *only* because it has these multi-request, implicit headers
> features, something is wrong with the abstractions.

   I think I agree.

Oleg.
-- 
 Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED]
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Creating dicts from dict subclasses

2006-12-13 Thread Walter Dörwald 
Guido van Rossum wrote:
> On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
>> I tried to reimplement weakref.WeakValueDictionary as a subclass of
>> dict. The test passes except for one problem: To compare results
>> test_weakref.py converts a weakdict to a real dict via dict(weakdict).
>> This no longer works because PyDict_Merge() does a PyDict_Check() on the
>> argument and then ignores all overwritten methods. (The old version
>> worked because UserDict.UserDict was used).
>>
>> The simplest solution is to replace the PyDict_Check() call with
>> PyDict_CheckExact(), but this might slow things down too much, because
>> the fallback code basically does:
>>
>>for key in iter(arg.keys()):
>>   self[key] = arg.__getitem__(key)
>>
>> Why can't we use:
>>
>>for key in iter(arg):
>>   self[key] = arg.__getitem__(key)
>>
>> instead?
> 
> The only reason I can think of is backwards compatibility: not all
> "mappings" created pre-2.2 would support iteration. Maybe you could
> check for a tp_iter slot and if non-NULL use the latter otherwise use
> the original fallback?

This doesn't seem to work. It breaks test_update() in test_dict.py which
does this:

d = {}
class SimpleUserDict:
def __init__(self):
self.d = {1:1, 2:2, 3:3}
def keys(self):
return self.d.keys()
def __getitem__(self, i):
return self.d[i]
d.update(SimpleUserDict())
self.assertEqual(d, {1:1, 2:2, 3:3})

This fails with

KeyError: 0

because SimpleUserDict doesn't implement __iter__, so it gets an
iterator implementation via __getitem__.

So maybe this only makes sense for Python 3.0 where we can demand that
dict-like classes implement __iter__?

Servus,
   Walter

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Creating dicts from dict subclasses

2006-12-13 Thread Guido van Rossum 
On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
> > On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> >> I tried to reimplement weakref.WeakValueDictionary as a subclass of
> >> dict. The test passes except for one problem: To compare results
> >> test_weakref.py converts a weakdict to a real dict via dict(weakdict).
> >> This no longer works because PyDict_Merge() does a PyDict_Check() on the
> >> argument and then ignores all overwritten methods. (The old version
> >> worked because UserDict.UserDict was used).
> >>
> >> The simplest solution is to replace the PyDict_Check() call with
> >> PyDict_CheckExact(), but this might slow things down too much, because
> >> the fallback code basically does:
> >>
> >>for key in iter(arg.keys()):
> >>   self[key] = arg.__getitem__(key)
> >>
> >> Why can't we use:
> >>
> >>for key in iter(arg):
> >>   self[key] = arg.__getitem__(key)
> >>
> >> instead?
> >
> > The only reason I can think of is backwards compatibility: not all
> > "mappings" created pre-2.2 would support iteration. Maybe you could
> > check for a tp_iter slot and if non-NULL use the latter otherwise use
> > the original fallback?
>
> This doesn't seem to work. It breaks test_update() in test_dict.py which
> does this:
>
> d = {}
> class SimpleUserDict:
> def __init__(self):
> self.d = {1:1, 2:2, 3:3}
> def keys(self):
> return self.d.keys()
> def __getitem__(self, i):
> return self.d[i]
> d.update(SimpleUserDict())
> self.assertEqual(d, {1:1, 2:2, 3:3})
>
> This fails with
>
> KeyError: 0
>
> because SimpleUserDict doesn't implement __iter__, so it gets an
> iterator implementation via __getitem__.
>
> So maybe this only makes sense for Python 3.0 where we can demand that
> dict-like classes implement __iter__?

Ah, right. But I think you should still use PyDict_CheckExact, and
slow fallbacks be damned. (I guess you could look for iterkeys first.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Open CPython VM for type information

2006-12-13 Thread Kay Schluehr 
Brett Cannon schrieb:

>
>
> On 12/12/06, *Kay Schluehr* <[EMAIL PROTECTED] 
> > wrote:
>
> Dear Python developers,
>
> while the JVM is opened to support dynamically typed languages [1] I
> wonder if the CPyVM could not show more openness to statically typed
> languages? Hereby I don't think that much about arbitrary
> languages for
> the CPyVM but sublanguages like RPython which are "static enough"
> to be
> compilable into extension modules [2].
>
> However creating C extension modules from Python code is not
> really the
> standard way I do think Python source should be handled. Instead I
> think
> about a more direct VM support like the JVM does, that provides
> opcodes
> like iload, fstore etc. Besides this I don't expect one canonical
> approach for these type information to be feeded into the CPyVM. There
> are type inferencer like rtyper for PyPy, Psycos continous bytecode
> monitoring and also other approaches are possible that use type
> feedback. Finally optional type declarations are also an issue;
> however
> I would like to see language design discussions being separated ( ->
> Python 3000 ) and focus on general APIs for compiler extensions /
> plugs
> that deal with type information ( the architecture ) and VM
> implementation issues.
>
> Thanks, Kay
>
> [1] http://jcp.org/en/jsr/detail?id=292
> [2] http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html
>
>
> The Python VM is not designed to be general-purpose, period.  Nor does 
> anyone on python-dev, to my knowledge, have any interest in putting 
> the time and energy into making it general.  I am quite happy with 
> letting the Parrot folk handle this goal.  This means that the 
> maintenance cost of adding more opcodes is not worth it.
>
> But a bigger issue with me is the performance hit.  Every opcode will 
> make the eval loop larger.  And it is known that the eval loop is 
> rather finicky in terms of size and performance.  While no hard 
> testing has been done, it has been assumed we have accidentally hit 
> the sweet spot in terms of CPU cache size.
>
> An important thing to keep in mind when comparing Python's VM to 
> Java's is that Java works (somewhat) for dynamic languages because 
> people are motivated enough to shoehorn languages into it by just 
> assuming everything is just an object and tossing out the type 
> checking inherent to Java.  But Java has not, to my knowledge, grown 
> any great features yet to make dynamic languages work better.  I know 
> about JSR 292, but that is not in yet and adding just one opcode is 
> not that expensive to the JVM.  But compare the cost of adding a slew 
> of typed opcodes to Python compared to the JVM adding just one is not 
> really equal.
>
> In other words, I wouldn't count on this ever happening.  =)  The best 
> you could do is try to propose some direct mechanism for adding 
> support to the eval loop to dispatch to some function for unrecognized 
> opcodes, but I wouldn't count on that being accepted either.
>
> -Brett

Brett, thanks for the short answer. Although it is negative regarding my 
attempt, it is also important to know which ideas are outlawed and where 
one steps into a no-go area concerning the evolving language standard 
and its main implementation.

Regards 

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Creating dicts from dict subclasses

2006-12-13 Thread Armin Rigo 
Hi Walter,

On Wed, Dec 13, 2006 at 05:57:16PM +0100, Walter D?rwald wrote:
> I tried to reimplement weakref.WeakValueDictionary as a subclass of
> dict. The test passes except for one problem: To compare results
> test_weakref.py converts a weakdict to a real dict via dict(weakdict).
> This no longer works because PyDict_Merge() does a PyDict_Check() on the
> argument and then ignores all overwritten methods. (The old version
> worked because UserDict.UserDict was used).

This is an instance of a general problem in Python: if you subclass a
built-in type, then your overridden methods may or may not be used in
various situations.  In this case you might have subtle problems with
built-in functions and statements that expect a dict and manipulate it
directly, because they will see the underlying dict structure.  It is
also quite fragile: e.g. if a future version of CPython adds a new
method to dicts, then your existing code will also grow the new method
automatically - but as inherited from 'dict', which produces quite
surprizing results for the user.

>for key in iter(arg.keys()):
>   self[key] = arg.__getitem__(key)
> 
> Why can't we use:
> 
>for key in iter(arg):
>   self[key] = arg.__getitem__(key)

The latter would allow 'arg' to be a sequence instead of a mapping.  It
may even not crash but produce nonsense instead, e.g. if 'arg' is a list
of small integers.  Moreover there are multiple places in the code base
that assume that mappings are "something with a 'keys' and a
'__getitem__'", so I suppose any change in that should be done
carefully.


A bientot,

Armin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com