Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: > I've seen people using everything from zip to touch, either out of > laziness or out of the fact it wouldn't work very well in python, this > zip case is a good example. so based on a limitation in one library, and some random code you've seen on the internet, you're makin

Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
Fredrik Lundh wrote: Hari Sekhon wrote: I take it that it's still a work in progress to be able to pythonify everything, and until then we're just gonna have to rely on shell and those great C coded coreutils and stuff like that. Ok, I'm rather fond of Bash+coreutils, highest r

Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: > I take it that it's still a work in progress to be able to pythonify > everything, and until then we're just gonna have to rely on shell and > those great C coded coreutils and stuff like that. Ok, I'm rather fond > of Bash+coreutils, highest ratio of code lines to work I'v

Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
Fredrik Lundh wrote: Hari Sekhon wrote: Is it me or is having to use os.system() all the time symtomatic of a deficiency/things which are missing from python as a language? it's you. I take it that it's still a work in progress to be able to pythonify everything

Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: > Is it me or is having to use os.system() all the time symtomatic of a > deficiency/things which are missing from python as a language? it's you. -- http://mail.python.org/mailman/listinfo/python-list

Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
On 20/05/06, Bruno Desthuilliers <[EMAIL PROTECTED]> wrote: Roger Miller a écrit :> The basic problem is that the zipfile interface only reads and writes> whole files, so it may perform poorly or fail on huge files.   At one> time I implemented a patch to allow reading files in chunks. However I >

Re: memory error with zipfile module

2006-05-19 Thread Bruno Desthuilliers
Roger Miller a écrit : > The basic problem is that the zipfile interface only reads and writes > whole files, so it may perform poorly or fail on huge files. At one > time I implemented a patch to allow reading files in chunks. However I > believe that the current interface has too many problems

Re: memory error with zipfile module

2006-05-19 Thread Roger Miller
The basic problem is that the zipfile interface only reads and writes whole files, so it may perform poorly or fail on huge files. At one time I implemented a patch to allow reading files in chunks. However I believe that the current interface has too many problems to solve by incremental patching,

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
Sion Arrowsmith wrote: > Hari Sekhon <[EMAIL PROTECTED]> wrote: (snip) >>The python zipfile module is obviously broken... > > This isn't at all obvious to me. zipfile.read() does not seem to take full advantage of zlib's decompressobj's features. This could perhaps be improved (left as an exerci

Re: memory error with zipfile module

2006-05-19 Thread Sion Arrowsmith
bruno at modulix <[EMAIL PROTECTED]> wrote: >http://mail.zope.org/pipermail/zope/2004-October/153882.html >""" >MemoryError is raised by Python when an underlying (OS-level) allocation >fails. >(...) >Normally this would mean that you were out of even virtual memory >(swap), but it could also be a

Re: memory error with zipfile module

2006-05-19 Thread Sion Arrowsmith
Hari Sekhon <[EMAIL PROTECTED]> wrote: >import zipfile >zip=zipfile.ZipFile('d:\somepath\cdimage.zip') >zip.namelist() >['someimage.iso'] [ ... ] >B) content=zip.read('someimage.iso') > >Traceback (most recent call last): > File "", line 1, in ? > File "D:\u\Python24\lib\zipfile.py", line 357

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
[EMAIL PROTECTED] wrote: > Take a look at the pywin32 extension, which I believe has some lower > level memory allocation and file capabilities that might help you in > this situation. But then the solution would not be portable, which would be a shame since the zlib module (on which ZipFile reli

Re: memory error with zipfile module

2006-05-19 Thread [EMAIL PROTECTED]
Take a look at the pywin32 extension, which I believe has some lower level memory allocation and file capabilities that might help you in this situation. If I'm completely wrong, someone please tell me XD. Of course, you could just make the read() a step process, reading, O lets say 8192 bytes at

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
Hari Sekhon wrote: > I do > > import zipfile > zip=zipfile.ZipFile('d:\somepath\cdimage.zip') > zip.namelist() > ['someimage.iso'] > > then either of the two: > > A) file('someimage.iso','w').write(zip.read('someimage.iso')) > or > B) content=zip.read('someimage.iso') > > but both result in

Re: memory error with zipfile module

2006-05-19 Thread Ganesan Rajagopal
> Hari Sekhon <[EMAIL PROTECTED]> writes: > Traceback (most recent call last): > File "", line 1, in ? > File "D:\u\Python24\lib\zipfile.py", line 357, in read > bytes = dc.decompress(bytes) > MemoryError Looks like the .iso file is huge. Even if it's only a CD image (approx 650MB), r