Hi all, Started a project year ago with hard goals in mind : Developing a game server which can handle thousands of clients simultaneously.System must be stable, scalable, efficient, and if the client need to migrate the server to another OS, this will be a matter of time, if possible without changing any code. Also, the system should be controlled via database operations, meaning some specific DB changes should be determined by the system on-the-fly and necassary processing should be done at runtime(Client timeout values, adding new game lobbies..etc..) And all of this should be a one-man project dues to the fact that the project is a volunteer work, thus is separate from our ordinary jobs, there is no budget:). This was 8 months ago, and I still remember we are talking with my colleguae to decide which programming language is better for our requirements. We wrote the tradeoffs of different languages on board and compare our objectives against them by giving numbers. I remember, the last three standing programming languages are: C++, Java, and Python. We decided that code maintability and stability is cheaper than native code efficiency and also easy porting is important. C++ is only successfull on efficiency but maintaining the code, even if we use 3rd party libraries is a nightmare, especially in a massive multithreaded project like a game server. Also, we decided a game server's bottleneck is on network I/O other than the actual processing which means there should be not so much performance degradation between C++ and an interpreted language. So the final is between Java and Python. I know Java has been adding very good optimizations for interpreter efficiency over the years(JIT compiling..etc.) And I remember somewhere reading that Java is 10 times faster than Python. But Python, on the other hand is a perfect language and very simple which means the project may be completed sooner and efficiency again will be not a so much problem as the bottleneck should be on network I/O if the server is coded correctly. Although we have made this decision, we still had some doubts but we move on with Python. I don't know Python at the time and only coded few simple projects with it. I started with reading python socket/threading tutorials, then I downloaded and read actual Python code to get deeper and deeper. This was 10 months ago. Now we have been able to implement the game server with Python using MySQL as a backend DB and currently 5000 players (increasing) are playing multiplayer games with it with a system load of 0.4-0.8(see Unix getloadavg output) on a 1.8 Ghz Dual Xeon processor. Let me give some benchmarks, please note that one thing I learn through this journey is that testing/benchmarking a server software is really difficult and maybe in some conditions impossible thing to do correctly as inactive client connections may cause indeterministic behavior and also WAN latencies are very different than LAN latencies..etc. Anyway, we had to do a benchmark before going with this server on live. We installed a UnrealIRCd server which is also using select() as an I/O paradigm just as we are and supporting many of the features we have. This server is a IRC server written in C and being written since nearly 10 years. We connect thousands of clients and do simple messaging between them. I don't have an official test/benchmark result or a nice graph for this, however the system running UnrealIRCD is locked up when processing 4 Mb/sec of data, where our 6 month old Python server is locking up the system resources after 3.5 Mb/sec. We have done other tests including stressing server accept(), adding inactive connections but I am really not %100 confident about the test environments and do not want to give you false impression. But we decided that this 0.5 Mb factor is still not enough for us and we move on to inspect what is really causing the bottlenecks and how to optimize them. Here, to be honest, I cannot see very uch help from Python. Profiling/Optimization is the only place I am upset with Python. Here is the requirements for the profiler: I want to be able to enable/disable profiler tool on the fly without requiring any kind of restart and the profiler should be multithreaded (thus thread-safe). I decided that cProfiler and other profiler tools are in need of some updates to fulfill my reqırements, which makes me write my own profiler. It took me 3 days to come up with a profiler that satisfies my requirements, this profiler will profile only my code, meaning the server code to avoid performance degradation and timings will need not be nested-aware and timing precision is not very important (other than secs). I have implemented the profiler with the help of decorators and started profiling my code on-live. Here comes the another part of Python that is some kind of shady: optimization. After profiling the code, it turns out most of the time is spent on the following: 1) Selecting/Deselecting interest sets for select() 2) Unnecessary function calls. (for integrating some OOP concepts, I think we have overlooked the performance factor and there really are some functions which can be inlined but declared as a function) 3) Redundant try-except's in all over place(Again our fault to make the system stable, we have put some debug purposed-assert like try- excepts in the main server flow.)
After, looking on these subjects for 2 weeks, we came up with solutions for all of them, and we see that it is working approximately 5 times faster. This test is done in live, we run two server processes which are all running on a predefined CPU(means CPU affinity of the process is set before hand to avoid mis-interpreting system diagnose tools like top(1)). When same client number is connected to both servers, the second one is giving 5 times better values in top and ps tools. Also, the processing time (meaning the server loop except the select() call) is also 10 times faster than before. Just one note about optimizing Python code: do not optimize Python code based on your assumptions, just go and test if it really runs faster. I don't want to go to details of this hint, but believe me making Python code optimized may be very very tricky. It is then I decided to write up here this as a success story, as I am very newcomer to Python but come up with a nearly commercial product in a very short period of time and I don't think this is about my personal characteristics and intelligence or so:), as I am old enough to know/meet that there are much much more brilliant people than I am and they also have similar experiences with Python. So, one last note: every software project goes same tasks as above often much much more officially and carefully, I would suggest managers to see that just do not listen to the ordinary brain-washes. Python is a great choice for easy developing, easy debugging, easy maintaining and most importantly very very time-friendly. Of course there will be tasks .n which Python is suitable, but hey, if it Python is in the list, take it seriously. Thanks, Special thanks to people in this group who had answered my silly Python questions along the way. Sümer Cip -- http://mail.python.org/mailman/listinfo/python-list
