Re: [Python-Dev] [Python-checkins] r64407 - python/trunk/Doc/library/multiprocessing.rst
On Thu, Jun 19, 2008 at 05:16:38PM -0400, Jesse Noller wrote: > Where would that chapter end up (source-wise) I think a few of us > might have additional things to add ;) This would be Doc/library/ipc.rst. The chapter is 'Interprocess Communication and Networking'. Is anyone else finding it increasingly odd that subprocess, signal, socket/ssl, and syncore are in the same chapter? I'm tempted to move socket, ssl, asyncore+asynchat into a 'networking' chapter, and then also move SocketServer from the 'Internet Protocols' chapter into this new chapter. --amk ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API for gc.enable() and gc.disable()
+1 on a C API for enabling and disabling GC. I have several instances where I create a large number of objects non-cyclic objects where I see huge GC overhead (30+ seconds with gc enabled, 0.15 seconds when disabled). +1000 to fixing the garbage collector to be smart enough to self-regulate itself better. In the mean time, I use the following context manager to deal with the hotspots as I find them: class gcdisabled(object): ''' Conext manager to temporarily disable Python's cyclic garbage collector. The primary use is to avoid thrashing while allocating large numbers of non-cyclic objects due to an overly aggressive garbage collector behavior. Will disable GC if it is enabled upon entry and renable upon exit: >>> gc.isenabled() True >>> with gcdisabled(): ... print gc.isenabled() False >>> print gc.isenabled() True Will not reenable if GC was disabled upon entry: >>> gc.disable() >>> gc.isenabled() False >>> with gcdisabled(): ... gc.isenabled() False >>> gc.isenabled() False ''' def __init__(self): self.isenabled = gc.isenabled() def __enter__(self): gc.disable() def __exit__(self, type, value, traceback): if self.isenabled: gc.enable() ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API for gc.enable() and gc.disable()
Hi, Kevin Jacobs bioinformed.com> gmail.com> writes: > > +1 on a C API for enabling and disabling GC. I have several instances where I create a large number of objects non-cyclic objects where I see huge GC overhead (30+ seconds with gc enabled, 0.15 seconds when disabled). Could you try to post a stripped-down, self-contained example of such behaviour? Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API for gc.enable() and gc.disable()
On Fri, Jun 20, 2008 at 10:25 AM, Antoine Pitrou <[EMAIL PROTECTED]> wrote: > > Kevin Jacobs bioinformed.com> gmail.com> > writes: > > > > +1 on a C API for enabling and disabling GC. I have several instances > where > I create a large number of objects non-cyclic objects where I see huge GC > overhead (30+ seconds with gc enabled, 0.15 seconds when disabled). > > Could you try to post a stripped-down, self-contained example of such > behaviour? $ python -m timeit 'zip(*[range(100)]*5)' 10 loops, best of 3: 496 msec per loop $ python -m timeit -s 'import gc; gc.enable()' 'zip(*[range(100)]*5)' 10 loops, best of 3: 2.93 sec per loop Note that timeit cheats and disables GC by default. Attached is a less stripped down script to demonstrate the super-linear behavior for somewhat naively coded transpose operators. The output is listed below: FUNCTIONROWS COLUMNS GC ENABLEDGC DISABLED -- --- -- transpose_comp5 00.0. transpose_comp5 250.55350.3537 transpose_comp5 501.43590.6868 transpose_comp5 752.71481.0760 transpose_comp5 1003.80701.3936 transpose_comp5 1255.51841.7617 transpose_comp5 1507.88282.1308 transpose_comp5 1759.32792.5364 transpose_comp5 200 11.82482.7399 transpose_comp5 225 14.74363.1585 transpose_comp5 250 18.44523.5818 transpose_comp5 275 21.48563.8988 transpose_comp5 300 24.41104.3148 transpose_zip 5 00.0. transpose_zip 5 250.25370.0658 transpose_zip 5 500.83800.1324 transpose_zip 5 751.75070.1989 transpose_zip 5 1002.61690.2648 transpose_zip 5 1254.07600.3317 transpose_zip 5 1505.88520.4145 transpose_zip 5 1757.39250.5161 transpose_zip 5 200 10.07550.6708 transpose_zip 5 225 14.26980.7760 transpose_zip 5 250 16.72910.9022 transpose_zip 5 275 20.38331.0179 transpose_zip 5 300 24.55151.0971 Hope this helps, -Kevin import gc import time def transpose_comp(rows): return [ [ row[i] for row in rows ] for i in xrange(len(rows[0])) ] def transpose_zip(rows): return zip(*rows) def bench(func, rows, cols): gc.enable() gc.collect() data = [ range(cols) ]*rows t0 = time.time() func(data) t1 = time.time() gc.disable() func(data) t2 = time.time() gc.enable() return t1-t0,t2-t1 print 'FUNCTIONROWS COLUMNS GC ENABLEDGC DISABLED ' print '-- --- -- ' rows = 5 for func in [transpose_comp,transpose_zip]: for cols in range(0,301,25): enabled,disabled = bench(func, rows, cols) print '%-15s %10d %10d %12.4f %12.4f' % \ (func.func_name,rows,cols,enabled,disabled) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API for gc.enable() and gc.disable()
2008/6/20 Kevin Jacobs <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>: > On Fri, Jun 20, 2008 at 10:25 AM, Antoine Pitrou <[EMAIL PROTECTED]> > wrote: >> >> Kevin Jacobs bioinformed.com> gmail.com> >> writes: >> > >> > +1 on a C API for enabling and disabling GC. I have several instances >> > where >> I create a large number of objects non-cyclic objects where I see huge GC >> overhead (30+ seconds with gc enabled, 0.15 seconds when disabled). >> >> Could you try to post a stripped-down, self-contained example of such >> behaviour? > > $ python -m timeit 'zip(*[range(100)]*5)' > 10 loops, best of 3: 496 msec per loop > > $ python -m timeit -s 'import gc; gc.enable()' 'zip(*[range(100)]*5)' > 10 loops, best of 3: 2.93 sec per loop I remember that a similar issue was discussed some months ago: http://bugs.python.org/issue2607 In short: the gc is tuned for typical usage. If your usage of python is specific, use gc.set_threshold and increase its values. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r64407 - python/trunk/Doc/library/multiprocessing.rst
> Is anyone else finding it increasingly odd that subprocess, signal, > socket/ssl, and syncore are in the same chapter? I'm tempted to move > socket, ssl, asyncore+asynchat into a 'networking' chapter, and then > also move SocketServer from the 'Internet Protocols' chapter into this > new chapter. Sounds like you mean, 'Non-HTTP Networking'. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (06/13/08 - 06/20/08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1917 open (+28) / 13057 closed (+19) / 14974 total (+47) Open issues with patches: 581 Average duration of open issues: 712 days. Median duration of open issues: 1479 days. Open Issues Breakdown open 1892 (+28) pending25 ( +0) Issues Created Or Reopened (50) ___ Implement __format__ for Decimal 06/20/08 http://bugs.python.org/issue2110reopened benjamin.peterson patch Create the urllib package06/18/08 http://bugs.python.org/issue2885reopened brett.cannon Python 2.5.2 Windows Source Distribution missing Visual Studio 2 06/13/08 http://bugs.python.org/issue3105created MarkE speedup some comparisons 06/13/08 http://bugs.python.org/issue3106created pitrou patch memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linu 06/13/08 http://bugs.python.org/issue3107created hushp1pt Implicit exception chaining (PEP 3134) 06/13/08 CLOSED http://bugs.python.org/issue3108created pitrou patch test_multiprocessing seems fragile 06/14/08 http://bugs.python.org/issue3109created pitrou Multiprocessing package build problem on Solaris 10 06/14/08 http://bugs.python.org/issue3110created jnoller multiprocessing ppc Debian/ ia64 Ubuntu compilation error06/14/08 http://bugs.python.org/issue3111created jnoller implement PEP 3134 exception reporting 06/14/08 http://bugs.python.org/issue3112created benjamin.peterson patch Document exception chaining 06/14/08 http://bugs.python.org/issue3113created benjamin.peterson bus error on lib2to3 06/15/08 CLOSED http://bugs.python.org/issue3114reopened gvanrossum patch os.listdir randomly fails on occasions when it shouldn't 06/14/08 http://bugs.python.org/issue3115created philipspencer patch Fix quadratic behavior for marshal.dumps() when len>32Mb 06/15/08 CLOSED http://bugs.python.org/issue3116created rhettinger patch, patch segfault with (None,) as argument in a def/lambda06/15/08 CLOSED http://bugs.python.org/issue3117created santoemma test_math fails on 64bit 06/16/08 CLOSED http://bugs.python.org/issue3118created benjamin.peterson patch, 64bit pickle.py is limited by python's call stack 06/16/08 http://bugs.python.org/issue3119created habnabit patch subprocess module truncates handles on AMD64 06/16/08 http://bugs.python.org/issue3120created rupole test_urllibnet fails 06/16/08 CLOSED http://bugs.python.org/issue3121created cartman sys.getsizeof() gives an AttributeError for _s
Re: [Python-Dev] C API for gc.enable() and gc.disable()
On Fri, Jun 20, 2008 at 9:44 AM, Amaury Forgeot d'Arc <[EMAIL PROTECTED]> wrote: > 2008/6/20 Kevin Jacobs <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>: >> On Fri, Jun 20, 2008 at 10:25 AM, Antoine Pitrou <[EMAIL PROTECTED]> >> wrote: >>> >>> Kevin Jacobs bioinformed.com> gmail.com> >>> writes: >>> > >>> > +1 on a C API for enabling and disabling GC. I have several instances >>> > where >>> I create a large number of objects non-cyclic objects where I see huge GC >>> overhead (30+ seconds with gc enabled, 0.15 seconds when disabled). >>> >>> Could you try to post a stripped-down, self-contained example of such >>> behaviour? >> >> $ python -m timeit 'zip(*[range(100)]*5)' >> 10 loops, best of 3: 496 msec per loop >> >> $ python -m timeit -s 'import gc; gc.enable()' 'zip(*[range(100)]*5)' >> 10 loops, best of 3: 2.93 sec per loop > > I remember that a similar issue was discussed some months ago: > http://bugs.python.org/issue2607 > > In short: the gc is tuned for typical usage. If your usage of python > is specific, > use gc.set_threshold and increase its values. For very large bursts of allocation, tuning is no different from disabling it outright, and disabling is simpler/more reliable. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r64407 - python/trunk/Doc/library/multiprocessing.rst
On Fri, Jun 20, 2008 at 08:55:13AM -0700, Bill Janssen wrote: > > Is anyone else finding it increasingly odd that subprocess, signal, > > socket/ssl, and syncore are in the same chapter? I'm tempted to move > > socket, ssl, asyncore+asynchat into a 'networking' chapter, and then > > also move SocketServer from the 'Internet Protocols' chapter into this > > new chapter. > > Sounds like you mean, 'Non-HTTP Networking'. I don't think so -- SMTP, FTP, NNTP, and telnet have nothing to do with HTTP, but they're certainly Internet protocols. --amk ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r64407 - python/trunk/Doc/library/multiprocessing.rst
> On Fri, Jun 20, 2008 at 08:55:13AM -0700, Bill Janssen wrote: > > > Is anyone else finding it increasingly odd that subprocess, signal, > > > socket/ssl, and syncore are in the same chapter? I'm tempted to move > > > socket, ssl, asyncore+asynchat into a 'networking' chapter, and then > > > also move SocketServer from the 'Internet Protocols' chapter into this > > > new chapter. > > > > Sounds like you mean, 'Non-HTTP Networking'. > > I don't think so -- SMTP, FTP, NNTP, and telnet have nothing > to do with HTTP, but they're certainly Internet protocols. Sure, but there's typically only one protocol that's carried over any of them. Whereas with REST and XML-RPC and so forth, there's a whole cottage industry of using HTTP as a carrier, in much the same way that TCP is used at a lower level. But you're right: there's "Networking / Generic Support", "Networking / Specific Protocols", and "Networking / HTTP-based", or some such. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r64424 - in python/trunk: Include/object.h Lib/test/test_sys.py Misc/NEWS Objects/intobject.c Objects/longobject.c Objects/typeobject.c Python/bltinmodule.c
I thought there was a discussion of this earlier, and the idea was to leave the prior implementation, because that's how it's implemented in 3.0. bin() is a new feature in 2.6, so there's no particular need to make it work like hex() and oct(). Recall that in 3.0, __bin__, __oct__, and __hex__ don't exist. Instead, you use __index__ for integer conversions. That's how bin() worked in 2.6 until this checkin. But now that I look for it, I can't find the original discussion. raymond.hettinger wrote: Author: raymond.hettinger Date: Fri Jun 20 06:18:15 2008 New Revision: 64424 Log: Make bin() implementation parallel oct() and hex() so that int/long subclasses can override or so that other classes can support. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r64407 - python/trunk/Doc/library/multiprocessing.rst
A.M. Kuchling wrote: On Fri, Jun 20, 2008 at 08:55:13AM -0700, Bill Janssen wrote: Is anyone else finding it increasingly odd that subprocess, signal, socket/ssl, and syncore are in the same chapter? I'm tempted to move socket, ssl, asyncore+asynchat into a 'networking' chapter, and then also move SocketServer from the 'Internet Protocols' chapter into this new chapter. Sounds like you mean, 'Non-HTTP Networking'. I don't think so -- SMTP, FTP, NNTP, and telnet have nothing to do with HTTP, but they're certainly Internet protocols. Perhaps we need a split between "networking technologies" and "network-based applications". regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r64407 - python/trunk/Doc/library/multiprocessing.rst
On Jun 20, 2008, at 5:46 PM, Steve Holden wrote: Perhaps we need a split between "networking technologies" and "network-based applications". Perhaps that would help. I certainly see HTTP as being on the same layer as SMTP and the like, but application protocols that ride on top of HTTP are a different beast. -Fred -- Fred Drake ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of Issue 2331 - Backport parameter annotations
> I can help. I don't have a patch against the trunk but my first > revisions of the patch > for annotations did handle things like tuple parameters which are > relevant to 2.6. Ah yes, I forgot about nested parameters. I see that 53170 still has nested parameters, but they were removed at some later point. Do you think this revision (and the patch it checked in) is the best starting point, or are my chances better with the patches at http://bugs.python.org/issue1607548 ? I just learned that default argument values are not allowed for nested parameters (in any version of Python AFAIK). Is there any reason for this other than lack of tuits? Cheers, David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API for gc.enable() and gc.disable()
Le vendredi 20 juin 2008 à 17:44 +0200, Amaury Forgeot d'Arc a écrit : > In short: the gc is tuned for typical usage. If your usage of python > is specific, > use gc.set_threshold and increase its values. It's fine for people "in the know" who take the time to test their code using various gc parameters. But most people don't (I know I never did, and until recently I didn't even know the gc could have such a disastrous effect on performance). I don't think expecting people to tweak gc parameters when they witness performance problems is reasonable. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
