[EMAIL PROTECTED] wrote:
> It sounds very interesting and i will definitely take a deeper look
> but i was more considering something simpler as a learning exercise
> more that finding a package that should already do what i want to
> achieve.
The bulk of this post dissects (roughly) how Kamaelia handles this
internally since I figure that might also be useful if you're after
this as a learning exercise :-) (It should hopefully be clear enough
for this to be useful) At minimum it might go some way to explain why
people generally tend to suggest using an existing framework :-)
> I want to find the basic guidelines to write that kind of client/server
> architecture and make some tests of different solutions(socket+select
> or socket+thread or mix of them)
I think if you're doing socket & select it sounds easy/simple until you
start dealing with all the cases you need to to do it properly. You're
probably best looking at the python cookbook for a good initial starting
point.
> Your project seems very great and will try to read in to learn.
Kamaelia *might* still be useful here in some respects, since despite being
*effectively* event based, due to the use of generators code tends to look
linear.
The low level structure of the server breaks down as follows:
(I'm spending a bit of time here since it might be of general interest,
sorry if it isn't)
* Kamaelia.Internet.TCPServer is a simple component that sits waiting for
a user to connect.
It makes the server port, adds any user socket options, makes it non
blocking, binds it to the requested (if any) host and either to a user
defined or random port number. (latter is for testing)
It then registers the socket with the piece of code that's taking care
of select handling, and sits to wait for any notifications of activity
on the socket. (Which would mean for something acting as a server a new
connection).
It then performs a socket accept, and creates an object to handle the
raw (now connected) socket - a connected socket adaptor. It then
registers that socket and object with the piece of code that's taking
care of select handling. (So that the select handling can tell the
piece of code handling the raw socket to do some work on the socket)
code: http://tinyurl.com/pocm8
* Kamaelia.Internet.ConnectedSocketAdaptor is actually more complicated
than you might expect. The reason is because the socket is
non-blocking, and where-ever you would normally block you have
to maintain some sort of buffering in case the action fails.
This is more a symptom of non-blocking behaviour than it is about
Kamaelia.
Anyhow the basics of this is to take any data that is to be sent to the
socket, and add that to a sending queue. (this data will generally
come from the code handling the protocol)
Then if the select handling code has said that we can send/should data,
we take data from the send queue and send as much as we can until our
sending fails (after all, we're non-blocking, so we expect it to fail).
Then if the select handling code has said that there's data ready to
receive the component reads from the socket until an error (again we
expect an error due to non-blocking behaviour). This data is then
passed on to a location where a protocol handler can expect to take the
data and do what it likes with it.
The bulk of the code for the CSA actually revolves around resends, and
the necessary error handling. (It's also one of the oldest bits of code
in Kamaelia so possibly a little horrid to look at!)
code: http://tinyurl.com/nj8lk
* Kamaelia.Internet.Selector is also slightly more complex than you might
think. This is because whilst you tell select "let me know whenever any
of these sockets is ready for (reading|writing|exceptional)", it
actually cares about what you do *after* it's given you an answer.
For example if you don't read from all the things that are ready to
read, or don't write to all the things that are ready to write or
(worse) try asking it to do things with regard to closed or invalid
sockets, then you're just asking for trouble.
There's lots of ways round this and you can do something like:
looping:
(readables, writeables, excepts) = select( ...)
for readable in readables:
read from readable and do something
for writable in writeables:
write to writeables and do something
for exceptional in excepts:
handle exceptional
However then you add new problems - what about if you want to do a
database request as result of one? Render a image for another? Wait for
keyboard press? etc.
As a result that's why we use explicit message passing, though many
people use a reactor (or sometimes a pro-actor) pattern to handle
events. (if you're familiar with simulation it's very similar to an
event wheel approach).
For us we simply have a service (a selector) that sits an waits to be
told "send me a message when this is ready to do X". Since the thing
handling select can't know (in our system because we don't want any
accidental serialisation) whether the event (socket read to ready,
socket ready to write) has been handled, our selector removes the
socket from the read/write/exceptional set when it's been found to be
ready to do something.
This means that when the thing using has read from the socket it tells
the selector "can I have more please", and if it's written to the
socket has to tell the selector "I want to do more, but it bust - can
you tell me when it's working again".
This means the code managing the select loop and the socket sets the
select call operates on has to deal with all these issues. (You'd have
to anyway, but in some respects it's more explicit at the socket-level
in Kamaelia)
As a result you'll see explicit code adding sockets to "readers",
"writers", and "exceptionals", and for removing sockets from them when
an event's happened. This puts responsibility for making this work with
the code that cares - the code handling the socket activity directly.
The rationale for this is that it then also means that the same code
that's used for sockets can be used for file handles. (one of the
points of select on unix after all)
code: http://tinyurl.com/mfqyr
For completeness:
* Kamaelia.Chassis.ConnectedServer.SimpleServer takes the above
components and packages them up in an easy to use way. (meaning you can
avoid all the lowlevel hassle).
It's called a Chassis because like you add components onto a car
chassis (wheels, seats, doors, engine) in order to make a car, you need
to add someting to the SimpleServer to make a server. Specifically you
need to give it something that can create components to handle
connections.
As a result, the way SimpleServer works is whenever a new connection
comes in, it gets told about it by being passed a connected socket
adaptor by the tcp server. It then creates a protocol handler to talk
to the connected socket adaptor. As a result, any information that
comes in from the socket gets passed to the protocol handler. Any
information the protocol handler generates gets sent out the socket.
As a result the logic for the SimpleServer is:
* Create the TCP Server
* Loop
* When the TCP Server gives us a connected socket adaptor,
create a protocol handler (from the given function), and wire
the two together.
* If the connection dies (due to the protocol handler saying
"shutdown" or due to the socket dying) the everyone interested
finds out and propogates the change. (Meaning you don't get
dead file descriptors in the select call)
code: http://tinyurl.com/pts72
You'll note that the explanation of select itself is by far the longest
here! You may also find "man select_tut" interesting and useful if you're
doing this as a learning exercise.
If you're looking to do this more generally, you should really consider
using SocketServer or asyncore that come with python, or Twisted or Kamelia
if you want to run a production system.
Twisted is more well known and more deployed and as a result more battle
tested (meaning seasoned programmers would sensibly trust it more!), and
follows a more common approach for coding all this. (Boils down to something
similar though) As a result if hiring people to work on code is something
to think about Twisted operates is probably a better choice. Kamaelia is
however designed to be easy to pick up.
Finally nipping back to your example here:
> 5 clients that get time updated from the server
The code to handle on the server side is trivial.
----(start)----
import time
from Axon.ThreadedComponent import threadedcomponent
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ConnectedServer import SimpleServer
from Kamaelia.Util.Stringify import Stringify
from Kamaelia.Util.Backplane import *
class periodictime(threadedcomponent):
def main(self):
while 1:
time.sleep(0.1)
self.send(time.time(), "outbox")
Backplane("periodictime").activate()
Pipeline(periodictime(),
Stringify(), # Make suitable for network
PublishTo("periodictime")).activate()
def getTimeProtocol():
return SubscribeTo("periodictime")
SimpleServer(protocol=getTimeProtocol, port=1600).run()
----(finish)----
Telnet to 127.0.0.1 1600 to see the result here.
If you want to have the data source something from a client (eg event info
coming in on port 1599) with clients of the event source coming in from
elsewhere, this would change the server as follows:
----(start)----
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ConnectedServer import SimpleServer
from Kamaelia.Util.Backplane import *
Backplane("periodictime").activate()
def publishTime(): return PublishTo("periodictime")
def getTimeProtocol(): return SubscribeTo("periodictime")
SimpleServer(protocol=publishTime, port=1599).run()
SimpleServer(protocol=getTimeProtocol, port=1600).run()
----(finish)----
And the time (event) source would look like this:
----(start)----
import time
from Kamaelia.Internet.TCPClient import TCPClient
from Axon.ThreadedComponent import threadedcomponent
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Stringify import Stringify
class periodictime(threadedcomponent):
def main(self):
while 1:
time.sleep(0.1)
self.send(time.time(), "outbox")
perdiodictimeserver = "127.0.0.1"
Pipeline(periodictime(),
Stringify(), # Make suitable for network
TCPClient(perdiodictimeserver, 1599)).activate()
----(finish)----
Anyway, I hope the explanation of what's going on inside the core is useful
since in many respects if you're writing your own select handling loop
(which I would encourage you to do if you're learning about this!), the
basics of what you have to do stay the same. (check activity, clear, when
errors happen ask again, buffer data which needs to get sent, and decouple
everything as best as makes sense whilst trying to avoid accidental
serialisations).
The reason for the examples in the end is merely for completeness. (I'll
probably add these to our SVN distribution since the question does seem to
crop up fairly often generally speaking!)
If you're looking to do this in a production environment I'm personally an
advocate of learning what's going on in the core and then using an existing
library.
(The reason Kamaelia exists is because I wondered if there was an
alternative, potentially clearer way of writing these things, most
people would quite sensibly just use Twisted - especially given you
can buy a book on it! I personally think Kamaelia is cleaner, but then
I would think that :)
Have fun!
Michael
--
http://mail.python.org/mailman/listinfo/python-list
