"Frank Millman" <fr...@chagford.com> writes: > Hi all > > I am writing a multi-user business/accounting application. It is getting > rather complex and I am looking at how to, not exactly simplify it, but find > a way to manage the complexity. > > I have realised that it is logically made up of a number of services - > database service with connection to database > workflow engine for business processes > services manager to handle automated services, such as web services > client manager to service logins from client workstations > possibly others > > I have made a start by splitting some of these off into separate modules and > running them in their own threads. > > I am concerned about scalability if they are all running on the same > machine, so I am looking into how to enable these services to run on > separate servers if required.
Have you finished the application already? At my job we're still serving just over 1M+ web requests (a month), processing 15k+ uploads, and searching through over 5M+ database records a day. We're still running on 3 boxes. You can get a lot out of your machines before you have to think about the complex task of scaling/distributing. > My first thought was to look into Pyro. It seems quite nice. One concern I > had was that it creates a separate thread for each object made available by > the server. My database server creates separate objects for each instance of > a row read in from the database, and with multiple users running multiple > applications, with each one opening multiple tables, this could run into > hundreds, so I was not sure if that would work. It probably will work. Pyro is a very nice framework and one that I've built a few applications on. It has a lot of flexible design patterns available. Just look in the examples included with the distribution. > > Then I read that the multiprocessing module allows processes to be spread > across multiple servers. The documentation is not as clear as Pyro's, but it > looks as if it could do what I want. I assume it would use processes rather > than threads to make multiple objects available, but I don't know if there > is a practical limit. There is a theoretical limit to all of the resources on a machine. Threads don't live outside of that limit. They just have a speedier start-up time and are able to communicate with one another in a single process. It doesn't sound like that will buy you a whole lot in your application. You can spawn as many processes as you need. > > Then I thought that, instead of the database server exposing each object > remotely, I could create one 'proxy' object on the server through which all > clients would communicate, and it in turn would communicate with each > instance locally. > > That felt more managable, but then I thought - why bother with remote > objects at all? Why not just run a SocketServer on the database server, and > design a mini-protocol to allow clients to make requests and receive > results. This is a technology I am already comfortable with, as this is how > I handle client workstation logins. If I did go this route, I could apply > the same principle to all the services. Because unless you wrote your own database or are using some arcane relic, it should already have its own configurable socket interface? > > I don't have the experience to make an informed decision at this point, so I > thought I would see if there is any consensus on the best way to go from > here. Finish building the application. Do the benchmarks. Profile. Optimize. Find the clear boundaries of each component. Build an API along those boundaries. Add a network layer in front of the boundaries. Pyro is a good choice, twisted is also good. Roll your own if you think you can do better or it would fit your projects' needs. > Is there any particular benefit in using remote objects as opposed to > writing a SocketServer? Abstraction. Pyro is just an abstraction over an RPC mechanism. Nothing special about it. Twisted has libraries to do the same thing. Writing your own socket-level code can be messy if you don't do it right. > > Any advice will be much appreciated. > > Thanks > > Frank Millman Best of luck. -- http://mail.python.org/mailman/listinfo/python-list
