Hello,
I have searched a good deal about this topic and have not found
any good information yet. It seems that the people asking all want
something a bit different than what I want and also don't divulge much
about their intentions. I wish to improve the rate of data transfer
between two python programs on the same machine. The project in
question is a database server that I built (written in python) that
works in conjunction with another python script for searching which is
executed via Apache. Basically, I am serving a database but since the
database is large, it remains in the memory of a server program. Upon
submitting a search request, the search script is called by Apache.
This search script makes a connection to the server, sends over the
query and waits on the reply. Queries and records are represented by
dictionaries and the database is a list of dictionaries. The only
problem here is that for large returns, it takes more time and I'd
like to transmit the data. The part that is slowing the data
transmission down seems to be the fact that the data has to travel
across a socket connection to be put back together on the other side.
The server makes a list of dictionaries which represents the records
that match the given criteria and then sends a pickle string of that
object back to the client (the search script).
What I'd like to do is somehow take the returned results of the search
and send over a memory address through the socket and let the search
script read the results directly from the server's memory rather than
send it all through the sockets. There is one more solution which
would work just as well for this application but would be less useful
in general terms. The second method would involve some tricks with the
python CGI module. I am fairly ignorant of how Apache works with the
CGI module but here is what I'd like to do. I want to somehow let the
server print out to the user's browser instead of the search script in
order to cut out the time of sending the results over the socket. This
sounds like it is not possible but I would like to know in greater
detail how the CGI module and Apache work together such that the
'print' statements write out to the user's browser. Mainly, I'd like
to know if there is any kind of descriptor or ID that can be passed
and used by another process to print output to the user's browser
instead of the script that Apache invoked.
If there is not a good Pythonic way to do the above, I am open to
mixing in some C to do the job if that is what it takes. I apologize
if this topic has been brought up many times before but hopefully I
have stated my intentions clearly enough for those with a higher
knowledge of the topic to help. If the above are not possible but you
have a really good idea for zipping large amounts of data from one
program to another, I'd like to hear it.
Thanks to all who take the time to read my request and also those with
a response.
--
http://mail.python.org/mailman/listinfo/python-list
