Marko Rauhamaa <ma...@pacujo.net> wrote: > Queues are fine if you hermetically insulate your objects. IOW, you > group your objects in single-threaded pseudoprocesses that don't make > any direct method calls to each other. If you do that, you might as well > use real processes. That way, you can even naturally enforce your > insulation assumption against accidents.
No, 10,000 processes will not do. First, they use more resources than threads, at least on Windows. Second, if something fails, you have 10,000 worker processes to kill. You also risk flooding your system with zombies. So thanks, but no thanks. I would consider a small pool of processes to utilize all cores on the CPU, though. But each process would have hundreds or thousands of threads. .NET AppDomains is an attempt to solve this. Python do not support AppDomains in the interpreter. It would be nice if it did, though, e.g. a thread with an isolated interpreter instance (similar to tcl). But currently the Python interpreter is not designed for that, as it has states which are lobal to the process, not just the interpreter. If we insulate objects completely, we must also serialize (pickle) them before sending them off as messages on a queue. That can be a major bottleneck. At least it is in numerical computing with Python. Avoiding serialization is also an important consideration. But IPC itself is very fast, so the important part is packing a object into a byte string and unpacking, not the overhead involved in sending it. Unix domain sockets and Windows named pipes are close to a memcpy in performance, with very little overhead. Pickle on the other hand is "very slow". Serialization is hard ot get around if we use processes for multi-core CPUs, but in a pure multithread design it is not an issue. Sturla -- https://mail.python.org/mailman/listinfo/python-list
