The results look right! I did a rather large test and the checksum passed. I will hold off any speed ups as you suggested.
On Wed, Feb 23, 2011 at 8:37 PM, Rob Williscroft <r...@rtw.me.uk> wrote: > Rita wrote in > news:AANLkTi=88dcpm_kqrs2g620obsnxz0majubfwpeme...@mail.gmail.com in > gmane.comp.python.general: > > [Top post relocated] > > > On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft <r...@rtw.me.uk> > > wrote: > > > >> Rita wrote in > >> news:AANLkTi=w95gxosc1tkt2bntgjqys1cbmdnojhokq4...@mail.gmail.com in > >> gmane.comp.python.general: > >> > >> > > >> > When using wait() it works a bit better but not consistent > >> > def run(cmd): > >> > p=subprocess.Popen(cmd,stdout=subprocess.PIPE) > >> > rc=p.wait() > >> > print rc > >> > return p.stdout > >> > > >> > >> > When the output of cmd is a small ascii file it works perfectly > >> > fine, but when the file is large (more than 2MB) the process just > >> > waits for ever (I am guessing its blocking?). > >> > >> Your OS has supplied a pipe buffer of 2MB, its full and the prossess > >> is waiting until you read something from the pipe (i.e. > >> p.stdout.read()). > >> > >> > When I use the communicate call > >> > it works perfectly but my process is consuming way too much memory. > >> > > >> > Is there a better way to get my return code consistently > >> > efficiently and not take up so much memory? > >> > >> If you don't need the processes output then don't use the PIPE > >> argument. If you do you will need to read from p.stdout until the > >> process is complete, then get the return code: > >> > >> while True: > >> buf = p.stdout.read( 1024 ) > >> # do somthing with buf > >> if len( buf ) < 1024: > >> break > >> > >> rc = p.wait() > > > > > For extra points is there a way to speed up the p.stdout.read(bufsize) > > ? > > Its unlikely that is the problem, most likely you are reading all of the > 2MB OS pipe buffer and then having to wait for the programme you are > calling to produce some more output. > > Before trying to speed things up be sure to test with real work being > done to the output you recieve. > > If its still slow and you have multiple cores/processors then you will > need to put you read loop in a thread to speed it up. > > -- > Rob. > > -- > http://mail.python.org/mailman/listinfo/python-list > -- --- Get your facts first, then you can distort them as you please.--
-- http://mail.python.org/mailman/listinfo/python-list
