On Oct 22, 10:32 am, Andy <[EMAIL PROTECTED]> wrote: > Dear Python dev community, > > I'm CTO at a small software company that makes music visualization > software (you can check us out atwww.soundspectrum.com). About two > years ago we went with decision to use embedded python in a couple of > our new products, given all the great things about python. We were > close to using lua but for various reasons we decided to go with > python. However, over the last two years, there's been one area of > grief that sometimes makes me think twice about our decision to go > with python... > > Some background first... Our software is used for entertainment and > centers around real time, high-performance graphics, so python's > performance, embedded flexibility, and stability are the most > important issues for us. Our software targets a large cross section > of hardware and we currently ship products for Win32, OS X, and the > iPhone and since our customers are end users, our products have to be > robust, have a tidy install footprint, and be foolproof. Basically, > we use embedded python and use it to wrap our high performance C++ > class set which wraps OpenGL, DirectX and our own software renderer. > In addition to wrapping our C++ frameworks, we use python to perform > various "worker" tasks on worker thread (e.g. image loading andprocessing). > However, we require *true* thread/interpreter > independence so python 2 has been frustrating at time, to say the > least. Please don't start with "but really, python supports multiple > interpreters" because I've been there many many times with people. > And, yes, I'm aware of the multiprocessing module added in 2.6, but > that stuff isn't lightweight and isn't suitable at all for many > environments (including ours). The bottom line is that if you want to > perform independentprocessing (in python) on different threads, using > the machine's multiple cores to the fullest, then you're out of luck > under python 2. > > Sadly, the only way we could get truly independent interpreters was to > put python in a dynamic library, have our installer make a *duplicate* > copy of it during the installationprocess(e.g. python.dll/.bundle -> > python2.dll/.bundle) and load each one explicitly in our app, so we > can get truly independent interpreters. In other words, we load a > fresh dynamic lib for each thread-independent interpreter (you can't > reuse the same dynamic library because the OS will just reference the > already-loaded one). > > From what I gather from the python community, the basis for not > offering "real" muti-threaded support is that it'd add to much > internal overhead--and I couldn't agree more. As a high performance C > and C++ guy, I fully agree that thread safety should be at the high > level, not at the low level. BUT, the lack of truly independent > interpreters is what ultimately prevents using python in cool, > powerful ways. This shortcoming alone has caused game developers-- > both large and small--to choose other embedded interpreters over > python (e.g. Blizzard chose lua over python). For example, Apple's > QuickTime API is powerful in that high-level instance objects can > leverage performance gains associated with multi-threadedprocessing. > Meanwhile, the QuickTime API simply lists the responsibilities of the > caller regarding thread safety and that's all its needs to do. In > other words, CPython doesn't need to step in an provide a threadsafe > environment; it just needs to establish the rules and make sure that > its own implementation supports those rules. > > More than once, I had actually considered expending company resources > to develop a high performance, truly independent interpreter > implementation of the python core language and modules but in the end > estimated that the size of that project would just be too much, given > our company's current resources. Should such an implementation ever > be developed, it would be very attractive for companies to support, > fund, and/or license. The truth is, we just love python as a > language, but it's lack of true interpreter independence (in a > interpreter as well as in a thread sense) remains a *huge* liability. > > So, my question becomes: is python 3 ready for true multithreaded > support?? Can we finally abandon our Frankenstein approach of loading > multiple identical dynamic libs to achieve truly independent > interpreters?? I've reviewed all the new python 3 C API module stuff, > and all I have to say is: whew--better late then never!! So, although > that solves modules offering truly independent interpreter support, > the following questions remain: > > - In python 3, the C module API now supports true interpreter > independence, but have all the modules in the python codebase been > converted over? Are they all now truly compliant? It will only take > a single static/global state variable in a module to potentially cause > no end of pain in a multiple interpreter environment! Yikes! > > - How close is python 3 really to true multithreaded use? The > assumption here is that caller ensures safety (e.g. ensuring that > neither interpreter is in use when serializing data from one to > another). > > I believe that true python independent thread/interpreter support is > paramount and should become the top priority because this is the key > consideration used by developers when they're deciding which > interpreter to embed in their app. Until there's a hello world that > demonstrates running independent python interpreters on multiple app > threads, lua will remain the clear choice over python. Python 3 needs > true interpreter independence and multi-threaded support!
What you describe, truly independent interpreters, is not threading at all: it is processes, emulated at the application level, with all the memory cost and none of the OS protections. True threading would involve sharing most objects. Your solution depends on what you need: * Killable "threads" -> OS processes * multicore usage (GIL removal) -> OS processes or alternative Python implementations (PyPy/Jython/IronPython) * Sane shared objects -> safethread -- http://mail.python.org/mailman/listinfo/python-list
