Karthik Gurusamy wrote: > On Jul 1, 12:38 pm, dlomsak <[EMAIL PROTECTED]> wrote: [...] > > I have found the stop-and-go between two processes on the same machine > leads to very poor throughput. By stop-and-go, I mean the producer and > consumer are constantly getting on and off of the CPU since the pipe > gets full (or empty for consumer). Note that a producer can't run at > its top speed as the scheduler will pull it out since it's output pipe > got filled up. > But when both processes are in the memory of the same machine and they communicate through an in-memory buffer, what's to stop them from keeping the CPU fully-loaded (assuming they are themselves compute-bound)?
> When you increased the underlying buffer, you mitigated a bit this > shuffling. And hence saw a slight increase in performance. > > My guess that you can transfer across machines at real high speed, is > because there are no process swapping as producer and consumer run on > different CPUs (machines, actually). > As a concept that's attractive, but it's easy to demonstrate that (for example) two machines will get much better throughput using the TCP-based FTP to transfer a large file than they do with the UDP-based TFTP. This is because the latter protocol requires the sending unit to stop and wait for an acknowledgment for each block transferred. With FTP, if you use a large enough TCP sliding window and have enough content, you can saturate a link as ling as its bandwidth isn't greater than your output rate. This isn't a guess ... > Since the two processes are on the same machine, try using a temporary > file for IPC. This is not as efficient as real shared memory -- but it > does avoid the IPC stop-n-go. The producer can generate the multi-mega > byte file at one go and inform the consumer. The file-systems have > gone thru' decades of performance tuning that this job is done really > efficiently. > I'm afraid this comes across a bit like superstition. Do you have any evidence this would give superior performance? > > >> Thanks for the replies so far, I really appreciate you guys >> considering my situation and helping out. > > regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading ------------- -- http://mail.python.org/mailman/listinfo/python-list
