Re: [Python-Dev] ast-objects branch created

2005-12-06 Thread Nick Coghlan 
Neal Norwitz wrote:
> This is my understanding of the two approaches from what I've seen so
> far (Jeremy or Martin should correct me if I'm wrong).
> 
> With current arena impl:
>  * need to call PyArena_AddPyObject() for any allocated PyObject
>  * need to call PyArena_AddMallocPointer() for any malloc()ed memory
> (there are current no manual calls like this, all the calls are in
> generated code?)
> 
> With the PyObject imp:
>  * need to init all PyObjects to NULL
>  * need to Py_XDECREF() on exit
>  * need to goto error if there is any failure
> 
> Both impls have a bit more details, but those are the highlights. 
>>From what I've seen of both, the arena is easier to deal with even
> though it is different from the rest of python.  There is only one
> thing to remember.

As Fredrik pointed out a while back, the PyObject approach doesn't *have* to 
involve manual decref operations - PyObject's come with a ready made arena 
structure, in the form of PyList.

However, whether the automatic management is done with a list or with Jeremy's 
arena structure, the style is still different from most of CPython, and either 
way there's going to be a small learning curve associated with getting used to 
it.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] commit of r41586 - in python/trunk: Lib/SimpleXMLRPCServer.py Misc/NEWS

2005-12-06 Thread Guido van Rossum 
Happened to see this commit. What's the magic about 10MB? Is there any
understanding of what causes it to fail? What is the failure mode?
Could it just be fragmentation causing the malloc or realloc to fail?
Should we perhaps use a more conservative buffer size, e.g. 1MB or
even 8K (the international standard for I/O buffering :-)?

--Guido

On 12/4/05, andrew.kuchling <[EMAIL PROTECTED]> wrote:
> Author: andrew.kuchling
> Date: Sun Dec  4 16:36:57 2005
> New Revision: 41586
>
> Modified:
>python/trunk/Lib/SimpleXMLRPCServer.py
>python/trunk/Misc/NEWS
> Log:
> [Bug #792570] Under Windows, socket.read() seems to run into trouble when
> asked to read tens of megabytes of data.  On my Mac, it hits MemoryErrors
> when reading around 15Mb in one chunk.  The fix is to read the body in several
> parts, not as one big piece.
>
> It would be nice to fix the underlying socket.read() problem, too.
>
> 2.4 bugfix candidate.
>
>
> Modified: python/trunk/Lib/SimpleXMLRPCServer.py
> ==
> --- python/trunk/Lib/SimpleXMLRPCServer.py  (original)
> +++ python/trunk/Lib/SimpleXMLRPCServer.py  Sun Dec  4 16:36:57 2005
> @@ -422,8 +422,19 @@
>  """
>
>  try:
> -# get arguments
> -data = self.rfile.read(int(self.headers["content-length"]))
> +# Get arguments by reading body of request.
> +# We read this in chunks to avoid straining
> +# socket.read(); around the 10 or 15Mb mark, some platforms
> +# begin to have problems (bug #792570).
> +max_chunk_size = 10*1024*1024
> +size_remaining = int(self.headers["content-length"])
> +L = []
> +while size_remaining:
> +chunk_size = min(size_remaining, max_chunk_size)
> +L.append(self.rfile.read(chunk_size))
> +size_remaining -= len(L[-1])
> +data = ''.join(L)
> +
>  # In previous versions of SimpleXMLRPCServer, _dispatch
>  # could be overridden in this class, instead of in
>  # SimpleXMLRPCDispatcher. To maintain backwards compatibility,
>
> Modified: python/trunk/Misc/NEWS
> ==
> --- python/trunk/Misc/NEWS  (original)
> +++ python/trunk/Misc/NEWS  Sun Dec  4 16:36:57 2005
> @@ -451,6 +451,9 @@
>  - Bug #1222790: in SimpleXMLRPCServer, set the reuse-address and 
> close-on-exec
>flags on the HTTP listening socket.
>
> +- Bug #792570: SimpleXMLRPCServer had problems if the request grew too large.
> +  Fixed by reading the HTTP body in chunks instead of one big socket.read().
> +
>  - Bug #1110478: Revert os.environ.update to do putenv again.
>
>  - Bug #1103844: fix distutils.install.dump_dirs() with negated options.
> ___
> Python-checkins mailing list
> [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/python-checkins
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] commit of r41586 - in python/trunk: Lib/SimpleXMLRPCServer.py Misc/NEWS

2005-12-06 Thread M.-A. Lemburg 
Guido van Rossum wrote:
> Happened to see this commit. What's the magic about 10MB? Is there any
> understanding of what causes it to fail? What is the failure mode?
> Could it just be fragmentation causing the malloc or realloc to fail?
> Should we perhaps use a more conservative buffer size, e.g. 1MB or
> even 8K (the international standard for I/O buffering :-)?

Just as data point: I've been hitting problems much earlier than
with 10MB (unrelated to SimpleXMLRPCServer.py, this is experience
from doing plain socket communication). Even 65535 is too much for
some platforms (AIX at the time). Things got stable at around 64000
bytes.

> --Guido
> 
> On 12/4/05, andrew.kuchling <[EMAIL PROTECTED]> wrote:
> 
>>Author: andrew.kuchling
>>Date: Sun Dec  4 16:36:57 2005
>>New Revision: 41586
>>
>>Modified:
>>   python/trunk/Lib/SimpleXMLRPCServer.py
>>   python/trunk/Misc/NEWS
>>Log:
>>[Bug #792570] Under Windows, socket.read() seems to run into trouble when
>>asked to read tens of megabytes of data.  On my Mac, it hits MemoryErrors
>>when reading around 15Mb in one chunk.  The fix is to read the body in several
>>parts, not as one big piece.
>>
>>It would be nice to fix the underlying socket.read() problem, too.
>>
>>2.4 bugfix candidate.
>>
>>
>>Modified: python/trunk/Lib/SimpleXMLRPCServer.py
>>==
>>--- python/trunk/Lib/SimpleXMLRPCServer.py  (original)
>>+++ python/trunk/Lib/SimpleXMLRPCServer.py  Sun Dec  4 16:36:57 2005
>>@@ -422,8 +422,19 @@
>> """
>>
>> try:
>>-# get arguments
>>-data = self.rfile.read(int(self.headers["content-length"]))
>>+# Get arguments by reading body of request.
>>+# We read this in chunks to avoid straining
>>+# socket.read(); around the 10 or 15Mb mark, some platforms
>>+# begin to have problems (bug #792570).
>>+max_chunk_size = 10*1024*1024
>>+size_remaining = int(self.headers["content-length"])
>>+L = []
>>+while size_remaining:
>>+chunk_size = min(size_remaining, max_chunk_size)
>>+L.append(self.rfile.read(chunk_size))
>>+size_remaining -= len(L[-1])
>>+data = ''.join(L)
>>+
>> # In previous versions of SimpleXMLRPCServer, _dispatch
>> # could be overridden in this class, instead of in
>> # SimpleXMLRPCDispatcher. To maintain backwards compatibility,
>>
>>Modified: python/trunk/Misc/NEWS
>>==
>>--- python/trunk/Misc/NEWS  (original)
>>+++ python/trunk/Misc/NEWS  Sun Dec  4 16:36:57 2005
>>@@ -451,6 +451,9 @@
>> - Bug #1222790: in SimpleXMLRPCServer, set the reuse-address and 
>> close-on-exec
>>   flags on the HTTP listening socket.
>>
>>+- Bug #792570: SimpleXMLRPCServer had problems if the request grew too large.
>>+  Fixed by reading the HTTP body in chunks instead of one big socket.read().
>>+
>> - Bug #1110478: Revert os.environ.update to do putenv again.
>>
>> - Bug #1103844: fix distutils.install.dump_dirs() with negated options.
>>___
>>Python-checkins mailing list
>>[EMAIL PROTECTED]
>>http://mail.python.org/mailman/listinfo/python-checkins
>>
> 
> 
> 
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> ___
> Python-checkins mailing list
> [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/python-checkins

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 06 2005)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] commit of r41586 - in python/trunk: Lib/SimpleXMLRPCServer.py Misc/NEWS

2005-12-06 Thread A.M. Kuchling 
On Tue, Dec 06, 2005 at 07:47:06AM -0800, Guido van Rossum wrote:
> Happened to see this commit. What's the magic about 10MB? Is there any
> understanding of what causes it to fail? What is the failure mode?
> Could it just be fragmentation causing the malloc or realloc to fail?
> Should we perhaps use a more conservative buffer size, e.g. 1MB or
> even 8K (the international standard for I/O buffering :-)?

At least on my Mac, it was a malloc error (a message is printed to
stderr by the malloc implementation).  SimpleXMLRPCServer ends up
catching the MemoryError and keeps running.  I don't know why the
malloc fails.

--amk
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] c.l.p post on docs

2005-12-06 Thread A.M. Kuchling 
I just posted a lengthy message to comp.lang.python/python-list about
Python's docs; the title is "Documentation suggestions".  A short
summary of the post is: "The RefGuide is hard to read and hard to keep
updated.  Do we need a friendly language description?  If we do that,
should the existing RefGuide be abandoned or maintained?"  See the
post for the full argument.

(Hey, the post just showed up in Google Groups:
)

I suggest further discussion on this issue take place in c.l.py. 

--amk

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation about Python's GC, python-dev list messages referenced in Modules/gcmodule.c not reachable anymore

2005-12-06 Thread Neil Schemenauer 
On Tue, Dec 06, 2005 at 11:20:46AM +0100, Weber, Gregoire wrote:
> We're seriously evaluating Python for use in embedded realtime systems
> and need some informations about Pythons garbage collector.
> 
> What we're interested mostly in the runtime behaviour of the GC. The
> main question is:
> 
>   Does it interrupt the python interpreter to collect stuff or 
>   is collecting done more in the background (e.g. just incrementally 
>   collecting)? This is an important question for realtime systems with
>   well defined reaction times.

It does not run in the background.  One option would be to disable
the cyclic garbage collector and rely on the reference counting
alone.  In that case, you will need to be sure that your code does
not create reference cycles.  Unfortunately I suspect there is now
Python library code that requires the cyclic collector to be
running.

> Just point me to documentation if available.

At this point the best documentation is the gcmodule.c code itself
(assuming the Python library documentation and my web page are not
sufficient).  The main entry point is collect().

If you really want to look at those old mailing list messages, you
can go the the index and search through the subject titles:

http://www.python.org/pipermail/python-dev/2000-March/

Here are some starting points:

http://mail.python.org/pipermail/python-dev/2000-March/002385.html
http://mail.python.org/pipermail/python-dev/2000-March/002497.html

Note that most of discussion was related to handling finalizers
(e.g. __del__ methods) and is not relevant to your concerns.
 
  Neil
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast-objects branch created

2005-12-06 Thread Martin v. Löwis 
Nick Coghlan wrote:
> As Fredrik pointed out a while back, the PyObject approach doesn't *have* to 
> involve manual decref operations - PyObject's come with a ready made arena 
> structure, in the form of PyList.

That doesn't really work: PyList_Append (which you would have to use) 
duplicates the reference, so you would still have to decref it
explicitly.

Of course, you could do so right away, instead of doing it on exit.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation about Python's GC, python-dev list messages referenced in Modules/gcmodule.c not reachable anymore

2005-12-06 Thread Tim Peters 
[Weber, Gregoire]
>> We're seriously evaluating Python for use in embedded realtime systems
>> and need some informations about Pythons garbage collector.
...

[Neil Schemenauer]
> It does not run in the background.  One option would be to disable
> the cyclic garbage collector and rely on the reference counting
> alone.

Python-style refcounting isn't generally a good approach either when
real-time constraints must be met:  when a refcount on an object P
falls to 0, not only does the interpreter "pause" to reclaim P, but
also to reclaim all the objects that were reachable only from P.  For
example, after

def f():
dummy = xrange(1000)

f()

it's not just the `dummy` list object that's reclaimed when f exits,
it's also about 10 million integer objects.  Deeply nested lists and
tuples (etc) can provoke similar "burps".

> In that case, you will need to be sure that your code does
> not create reference cycles.  Unfortunately I suspect there is now
> Python library code that requires the cyclic collector to be
> running.

And in the core.  For example, new-style class objects are full of
cycles -- although it's unlikely most programs will create a large
number of new-style classes dynamically.

Are there any languages with gc that are suitable for real-time work? 
Probably not without a lot of effort specifically aimed at meeting
real-time constraints.  It's also generally true that the Python core
and libraries use algorithms with good expected-case behavior but
horrid worst-case behavior.  For most apps, that's a big win most of
the time; for real-time apps, that _can_ be disastrous.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation about Python's GC, python-dev list messages referenced in Modules/gcmodule.c not reachable anymore

2005-12-06 Thread Tim Peters 
[Tim Peters]
> ...
> For example, after
>
>def f():
>dummy = xrange(1000)
>
>f()
>
> it's not just the `dummy` list object that's reclaimed when f exits,
> it's also about 10 million integer objects.

Sorry, that example should have used "range" instead of "xrange". 
Using xrange, no integer objects are created ;-)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

2005-12-06 Thread Raymond Hettinger 
[Matthew F. Barnes]  Perhaps it would be a useful addition to the
itertools
> module then?
> 
> itertools.interruptable(iterable)

Any real-world use cases or compelling contrived examples?

ISTM, that the code calling it.stop() would already be in position to
break-out of the iteration directly or set a termination flag.  Instead
of:

it = itertools.interruptable(iterable):
for x in it:
. . .
if cond(x):
it.stop()

Why not write:

for x in iterable:
. . .
if cond(x):
break

If needed, the for-loop can have an else-clause for any processing
needed in the event of interruption.



Raymond 

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

2005-12-06 Thread Simon Wittber 
On 12/7/05, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> [Matthew F. Barnes]  Perhaps it would be a useful addition to the
> itertools
> > module then?
> >
> > itertools.interruptable(iterable)
>
> Any real-world use cases or compelling contrived examples?

I use something like this in the nanothreads module.

http://metaplay.dyndns.org:8081/svn/fibranet/fibranet/nanothreads.py

This feature is implemented in the Fibra class, in the end and kill
methods. It is used to stop an iterator from parts of the code which
are not part of the loop that is actually iterating the iterator.

This usage is specific to situations where generators are being used
as cooperative threads.

-Sw,
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

2005-12-06 Thread Raymond Hettinger 
[Matthew F. Barnes]  
> > > Perhaps it would be a useful addition to the itertools
> > > module then?
> > >
> > > itertools.interruptable(iterable)

[Raymond Hettinger]
> > Any real-world use cases or compelling contrived examples?

[Simon Wittber]
> I use something like this in the nanothreads module.
> 
> http://metaplay.dyndns.org:8081/svn/fibranet/fibranet/nanothreads.py
> 
> This feature is implemented in the Fibra class, in the end and kill
> methods. It is used to stop an iterator from parts of the code which
> are not part of the loop that is actually iterating the iterator.
> 
> This usage is specific to situations where generators are being used
> as cooperative threads.

Are there any generator specific needs that are not met by the PEP 342
implementation?  Given a choice between throw(), close(), and send(), I
would have thought that all the bases have been covered.


Raymond

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

2005-12-06 Thread Simon Wittber 
On 12/7/05, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
> Are there any generator specific needs that are not met by the PEP 342
> implementation?  Given a choice between throw(), close(), and send(), I
> would have thought that all the bases have been covered.

Agreed. When the new functionality in PEP 342 arrives, most of the
generator tricks in nanothreads.py will become redundant.

-Sw.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com