Thanks for the responses folks. I'm starting to think that there is merely an inefficiency in how I'm using the sockets. The expensive part of the program is definitely the socket transfer because I timed each part of the routine individually. For a small return, the whole search and return takes a fraction of a second. For a large return (in this case 21,000 records - 8.3 MB) is taking 18 seconds. 15 of those seconds are spent sending the serialized results from the server to the client. I did a little bit of a blind experiment and doubled the bytes on the client's socket.recv line. This improved the rate of transfer each time. The original rate when I was accepting 1024 bytes per recv took 47 seconds to send the 8.3 MB result. By doubling this size several times, I reduced the time to 18 seconds until doubling it further produced diminishing results. I was always under the impression that keeping the send and recv byte sizes around 1024 is a good idea and I'm sure that jacking those rates up is a lousy way to mitigate the transfer. It is also interesting to note that increasing the bytes sent per socket.send on the server side had no visible effect. Again, that was just a curious experiment.
What bothers me is that I am sure sending data over the local loopback address should be blazing fast. 8.3 MB should be a breeze because I've transferred files over AIM to people connected to the same router as me and was able to send hundreds of megabytes in less than a two or three seconds. With that said, I feel like something about how I'm send/recv-ing the data is causing lots of overhead and that I can avoid reading the memory directly if I can speed that up. I guess now I'd like to know what are good practices in general to get better results with sockets on the same local machine. I'm only instantiating two sockets total right now - one client and one server, and the transfer is taking 15 seconds for only 8.3MB. If you guys have some good suggestions on how to better utilize sockets to transfer data at the speeds I know I should be able to achieve on a local machine, let me know what you do. At present, I find that using sockets in python requires very few steps so I'm not sure where I could really improve at this point. Thanks for the replies so far, I really appreciate you guys considering my situation and helping out. -- http://mail.python.org/mailman/listinfo/python-list
