Dear Python dev community, I'm CTO at a small software company that makes music visualization software (you can check us out at www.soundspectrum.com). About two years ago we went with decision to use embedded python in a couple of our new products, given all the great things about python. We were close to using lua but for various reasons we decided to go with python. However, over the last two years, there's been one area of grief that sometimes makes me think twice about our decision to go with python...
Some background first... Our software is used for entertainment and centers around real time, high-performance graphics, so python's performance, embedded flexibility, and stability are the most important issues for us. Our software targets a large cross section of hardware and we currently ship products for Win32, OS X, and the iPhone and since our customers are end users, our products have to be robust, have a tidy install footprint, and be foolproof. Basically, we use embedded python and use it to wrap our high performance C++ class set which wraps OpenGL, DirectX and our own software renderer. In addition to wrapping our C++ frameworks, we use python to perform various "worker" tasks on worker thread (e.g. image loading and processing). However, we require *true* thread/interpreter independence so python 2 has been frustrating at time, to say the least. Please don't start with "but really, python supports multiple interpreters" because I've been there many many times with people. And, yes, I'm aware of the multiprocessing module added in 2.6, but that stuff isn't lightweight and isn't suitable at all for many environments (including ours). The bottom line is that if you want to perform independent processing (in python) on different threads, using the machine's multiple cores to the fullest, then you're out of luck under python 2. Sadly, the only way we could get truly independent interpreters was to put python in a dynamic library, have our installer make a *duplicate* copy of it during the installation process (e.g. python.dll/.bundle -> python2.dll/.bundle) and load each one explicitly in our app, so we can get truly independent interpreters. In other words, we load a fresh dynamic lib for each thread-independent interpreter (you can't reuse the same dynamic library because the OS will just reference the already-loaded one). >From what I gather from the python community, the basis for not offering "real" muti-threaded support is that it'd add to much internal overhead--and I couldn't agree more. As a high performance C and C++ guy, I fully agree that thread safety should be at the high level, not at the low level. BUT, the lack of truly independent interpreters is what ultimately prevents using python in cool, powerful ways. This shortcoming alone has caused game developers-- both large and small--to choose other embedded interpreters over python (e.g. Blizzard chose lua over python). For example, Apple's QuickTime API is powerful in that high-level instance objects can leverage performance gains associated with multi-threaded processing. Meanwhile, the QuickTime API simply lists the responsibilities of the caller regarding thread safety and that's all its needs to do. In other words, CPython doesn't need to step in an provide a threadsafe environment; it just needs to establish the rules and make sure that its own implementation supports those rules. More than once, I had actually considered expending company resources to develop a high performance, truly independent interpreter implementation of the python core language and modules but in the end estimated that the size of that project would just be too much, given our company's current resources. Should such an implementation ever be developed, it would be very attractive for companies to support, fund, and/or license. The truth is, we just love python as a language, but it's lack of true interpreter independence (in a interpreter as well as in a thread sense) remains a *huge* liability. So, my question becomes: is python 3 ready for true multithreaded support?? Can we finally abandon our Frankenstein approach of loading multiple identical dynamic libs to achieve truly independent interpreters?? I've reviewed all the new python 3 C API module stuff, and all I have to say is: whew--better late then never!! So, although that solves modules offering truly independent interpreter support, the following questions remain: - In python 3, the C module API now supports true interpreter independence, but have all the modules in the python codebase been converted over? Are they all now truly compliant? It will only take a single static/global state variable in a module to potentially cause no end of pain in a multiple interpreter environment! Yikes! - How close is python 3 really to true multithreaded use? The assumption here is that caller ensures safety (e.g. ensuring that neither interpreter is in use when serializing data from one to another). I believe that true python independent thread/interpreter support is paramount and should become the top priority because this is the key consideration used by developers when they're deciding which interpreter to embed in their app. Until there's a hello world that demonstrates running independent python interpreters on multiple app threads, lua will remain the clear choice over python. Python 3 needs true interpreter independence and multi-threaded support! Thanks, Andy O'Meara -- http://mail.python.org/mailman/listinfo/python-list
