Re: pickle.load() extremely slow performance

2009-03-23 Thread Jim Garrison
Steve Holden wrote: Jean-Paul Calderone wrote: On Mon, 23 Mar 2009 10:57:54 -0500, Jim Garrison wrote: Benjamin Peterson wrote: Terry Reedy udel.edu> writes: 3.1a1 is out and I believe it has the io improvements. Massive ones, too. It'd be interesting to see your results on the alpha. On

Re: pickle.load() extremely slow performance

2009-03-23 Thread Steve Holden
Jean-Paul Calderone wrote: > On Mon, 23 Mar 2009 10:57:54 -0500, Jim Garrison wrote: >> Benjamin Peterson wrote: >>> Terry Reedy udel.edu> writes: 3.1a1 is out and I believe it has the io improvements. >>> >>> Massive ones, too. It'd be interesting to see your results on the alpha. >> >> On

Re: pickle.load() extremely slow performance

2009-03-23 Thread Jean-Paul Calderone
On Mon, 23 Mar 2009 10:57:54 -0500, Jim Garrison wrote: Benjamin Peterson wrote: Terry Reedy udel.edu> writes: 3.1a1 is out and I believe it has the io improvements. Massive ones, too. It'd be interesting to see your results on the alpha. On 3.1a1 the unpickle step takes 2.4 seconds, an 1

Re: pickle.load() extremely slow performance

2009-03-23 Thread Jim Garrison
Benjamin Peterson wrote: Terry Reedy udel.edu> writes: 3.1a1 is out and I believe it has the io improvements. Massive ones, too. It'd be interesting to see your results on the alpha. On 3.1a1 the unpickle step takes 2.4 seconds, an 1875% improvement. Thanks. -- http://mail.python.org/mailm

Re: pickle.load() extremely slow performance

2009-03-21 Thread Benjamin Peterson
Terry Reedy udel.edu> writes: > > 3.1a1 is out and I believe it has the io improvements. Massive ones, too. It'd be interesting to see your results on the alpha. -- http://mail.python.org/mailman/listinfo/python-list

Re: pickle.load() extremely slow performance

2009-03-20 Thread Terry Reedy
Carl Banks wrote: The slow performance is most likely due to the poor performance of Python 3's IO, which is caused by (among other things) bad buffering strategy. It's a Python 3 growing pain, and is being rewritten. Python 3.1 should be must faster but it's not been released yet. 3.1a1 is o

Re: pickle.load() extremely slow performance

2009-03-20 Thread bearophileHUGS
Carl Banks: > The slow performance is most likely due to the poor performance of > Python 3's IO, which is caused by [...] My suggestion for the Original Poster is just to try using Python 2.x, if possible :-) Bye, bearophile -- http://mail.python.org/mailman/listinfo/python-list

Re: pickle.load() extremely slow performance

2009-03-20 Thread Carl Banks
On Mar 20, 5:26 pm, Jim Garrison wrote: > John Machin wrote: > > On Mar 21, 9:25 am, Jim Garrison wrote: > >> I'm converting a Perl system to Python, and have run into a severe > >> performance problem with pickle. > > >> One facet of the system involves scanning and loading into memory a > >> co

Re: pickle.load() extremely slow performance

2009-03-20 Thread Jim Garrison
Jim Garrison wrote: > John Machin wrote: [snip] >> Have you considered using cPickle instead of pickle? >> Have you considered using *ickle.dump(..., protocol=-1) ? > > I'm using Python 3 on Windows (Server 2003). According to the docs > >"The pickle module has an transparent optimizer (_pic

Re: pickle.load() extremely slow performance

2009-03-20 Thread Jim Garrison
John Machin wrote: > On Mar 21, 9:25 am, Jim Garrison wrote: >> I'm converting a Perl system to Python, and have run into a severe >> performance problem with pickle. >> >> One facet of the system involves scanning and loading into memory a >> couple of parallel directory trees containing OTO 10^4

Re: pickle.load() extremely slow performance

2009-03-20 Thread John Machin
On Mar 21, 9:25 am, Jim Garrison wrote: > I'm converting a Perl system to Python, and have run into a severe > performance problem with pickle. > > One facet of the system involves scanning and loading into memory a > couple of parallel directory trees containing OTO 10^4 files.  The > trees don't

Re: pickle.load() extremely slow performance

2009-03-20 Thread John Machin
On Mar 21, 9:25 am, Jim Garrison wrote: > I'm converting a Perl system to Python, and have run into a severe > performance problem with pickle. > > One facet of the system involves scanning and loading into memory a > couple of parallel directory trees containing OTO 10^4 files.  The > trees don't

pickle.load() extremely slow performance

2009-03-20 Thread Jim Garrison
I'm converting a Perl system to Python, and have run into a severe performance problem with pickle. One facet of the system involves scanning and loading into memory a couple of parallel directory trees containing OTO 10^4 files. The trees don't change during development/testing and the scan tak