[issue10243] Packaged Pythons
New submission from Max Skaller : Not sure if this is a bug or not. I am unable to find libpython.so for Python3 on either my Mac or Ubuntu. Perhaps this is a packaging fault, however some documentation in the Wiki suggests otherwise. It appears the builders have reverted to an archaic linkage pattern which I helped to get rid of (lets see, perhaps a decade ago?). Python 2.6, for example, does ship a shared library. Python must be shipped with code linked as follows, I will explain below why, but first the list: 1) The mainline (C main() function) must be a stub which calls the real mainline, which is located in libpython. 2) The mainline MUST be compiled with C++ not C. 3) All extension libraries and add-ons to Python provided as shared libraries must be explicitly linked against libpython. In particular it is NOT acceptable for any extension or shared library component to expect to find its symbols in the host application executable as the Wiki documentation seems to suggest (in a section which explains a bit of a workaround for OSX frameworks). Now the reason it MUST be this way. First, any C++ code which is to be linked into an application, either statically, dynamically at load time, or under program control at run time, may require certain stuff to be in place (RTTI, streams, exception handling stuff, or whatever) which can only be put in place in the initialisation of the main application. Although the details are platform specific, it is simply not safe to permit C++ extension modules unless this step is taken. Legacy or embedded systems may have to make do with a C mainline, and systems which don't support dynamic loading can also do without C++ compilation provided the pre-loaded extensions are all C. On most major platforms, however, a C++ driver stub is required. The second issue is also quite simple. It is quite incorrect in a modern computing environment to assume an *application* will be hosting the python interpreter directly. It is not only possible, it is in fact the case for my project, that were the Python interpreter to be called, it would from a shared library loaded under program control at run time. Such an interpreter cannot be loaded at all if it isn't present in a library: it either has to be statically linked into the shared library making the call, with some ugly linker switches to make sure no symbols are dropped, or it has to be loaded dynamically. The latter case is the only viable option if the run time linker is unable to share symbols to a loaded application, and even if that is possible and can be arranged it is not likely to work so well if multiple shared libraries try to do it. Similarly, even if you managed to load it somehow, any dynamically loaded extensions may or may not be able to find the symbols. The ONLY reliable way to ensure extensions can find libpython symbols is to link them against libpython. In fact, the mainline application should not only contain NO libpython symbols specifically to disable this wrong practice and break any bad extensions that rely on it, it should also, as explained, contain exactly one reference to libpython, which calls Python with argv, argc[] as if it were the mainline. Just as motivation here: my product is an ultra-high performance programming language with a special construction to allow Python C-extension modules to be built. Its target is usually a shared library and that library is produced by first generating C++ and then compiling it with your native C++ compiler. For a generated program to call Python interpreter, it HAS to be available in a shared library, and for any extension modules that interpreter loads, they HAVE to get their symbols from that shared library, and, if the generated program is itself a Python module, then if that module is to be loaded from any C extension, including itself or some other extension, it HAS to be linked against libpython which HAS to be loaded dynamically by the loaded. The unfortunate downside of this is that it is NOT POSSIBLE to have a huge statically linked Python executable which just loads C extensions and nothing else happens. If you're loading any C extensions dynamically libpython must be loaded dynamically too. Just to be clear: I can easily build it the way I want it but this will not solve my problem, which is to support clients who wish to use my product to generate high performance Python callable modules which will just work "out of the box" with existing Python code. In particular, replacing some slow modules with optimised ones would be more or less entirely transparent .. except that at the moment it could only work with Python 2.x since Python 3.x shipments don't seem to have any shared libpython included (and I just changed my compiler to support Python 3 modules instead of Python 2). -- components: Installation messages: 119963 nosy: Max.Skaller pri
[issue10243] Packaged Pythons
Max Skaller added the comment: On Sat, Oct 30, 2010 at 6:40 PM, Martin v. Löwis wrote: It may be there is none. You need to read the bit where I explain that I am not building Python, I'm grabbing pre-made packages, for OSX and for Ubuntu. The problem is that these packages don't seem to supply a dynamic link version. My problem cannot be solved by telling me I can build Python myself with the --enable-shared switch, because I am not the client. I am a vendor supplying a tool that can generate Python shared libraries which cannot run unless the CLIENT has a shared library version of libpython. So you're telling me to tell THEM to build Python with --enable-shared switch which is out of the question for many of them, who may, for example, be programmers working in a shop where they do not have the ability to change the system installed by their system admin. So the problem is that the *packagers* are not supplying the dynamic lib. Surely that is not the Python dev's issue directly, but it IS an issue the Python dev's can do something about, by talking to the packagers. Anyhow, I will give up. I can't test the feature of the compiler I have implemented because I don't have a libpython.so and I have no intention of building one, because I can't expect all those Python users out there to do it either. It seems you really don't understand the requirements for dynamic linking: my application code is exclusively in a dlopen()d shared library, so if it is used AS a python module or it wants itself to USE a Python extension module OR Python interpreter itself, it cannot do so. The top level application is a fixed mainline which does not include libpython.a or any such symbols. It's a basic design principle, Meyer called it "the principle of explicit interfaces" which means: if you depend on something make the dependency explicit. Extension modules which do not *explicitly* link against libpython break this rule. -- Added file: http://bugs.python.org/file19480/unnamed ___ Python tracker <http://bugs.python.org/issue10243> ___On Sat, Oct 30, 2010 at 6:40 PM, Martin v. Löwis <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Martin v. Löwis <mailto:mar...@v.loewis.de";>mar...@v.loewis.de> added the comment: > Python 2.6, for example, does ship a shared library. I fail to see the bug in this report.It may be there is none. You need to read the bit where I explain that I am not building Python, I'm grabbing pre-made packages, for OSX and for Ubuntu. The problem is that these packages don't seem to supply a dynamic link version.My problem cannot be solved by telling me I can build Python myself withthe --enable-shared switch, because I am not the client. I am a vendor supplying a tool that can generate Python shared libraries which cannot run unless the CLIENT has a shared library version of libpython. So you're telling me to tell THEM to build Python with --enable-shared switch which is out of the question for many of them, who may, for example, be programmers working in a shop where they do not have the ability to change the system installed by their system admin.  So the problem is that the *packagers* are not supplying the dynamic lib.Surely that is not the Python dev's issue directly, but it IS an issue the Python dev's can do something about, by talking to the packagers. Anyhow, I will give up. I can't test the feature of the compiler I have implementedbecause I don't have a libpython.so and I have no intention of building one,because I can't expect all those Python users out there to do it either. It seems you really don't understand the requirements for dynamic linking:my application code is exclusively in a dlopen()d shared library, so if it isused AS a python module or it wants itself to USE a Python extension module OR Python interpreter itself, it cannot do so.The top level application is a fixed mainline which does not includelibpython.a or any such symbols.It's a basic design principle, Meyer called it "the principle of explicit interfaces" which means: if you depend on something make the dependency explicit.Extension modules which do not *explicitly* link against libpython breakthis rule. ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10243] Packaged Pythons
Max Skaller added the comment: On Thu, Nov 4, 2010 at 5:19 PM, Ned Deily wrote: > > Ned Deily added the comment: > > For what it's worth, the python.org installers for Mac OS X do include a > libpython shared library. As of Python 2.7 (and 3.2), the installer > includes a symlink to make it easier to find: > > $ cd /Library/Frameworks/Python.framework/Versions/2.7/lib > $ ls -l libpython2.7.dylib > Ok.. so why is it called Python instead of Python.dylib? /Library/Frameworks/Python.framework>file Python Python: broken symbolic link to Versions/Current/Python /Library/Frameworks/Python.framework/Versions/3.1>file Python Python: Mach-O universal binary with 2 architectures Python (for architecture ppc): Mach-O dynamically linked shared library ppc Python (for architecture i386): Mach-O dynamically linked shared library i386 Hmm .. i386? Oh dear, I'm running Snow Leopard and I generate 64 bit code. -- Added file: http://bugs.python.org/file19498/unnamed ___ Python tracker <http://bugs.python.org/issue10243> ___On Thu, Nov 4, 2010 at 5:19 PM, Ned Deily <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Ned Deily <mailto:n...@acm.org";>n...@acm.org> added the comment: For what it's worth, the http://python.org"; target="_blank">python.org installers for Mac OS X do include a libpython shared library. Â As of Python 2.7 (and 3.2), the installer includes a symlink to make it easier to find: $ cd /Library/Frameworks/Python.framework/Versions/2.7/lib $ ls -l libpython2.7.dylibOk.. so why is it called Python instead of Python.dylib?/Library/Frameworks/Python.framework>file PythonPython: broken symbolic link to Versions/Current/Python /Library/Frameworks/Python.framework/Versions/3.1>file PythonPython: Mach-O universal binary with 2 architecturesPython (for architecture ppc):Â Mach-O dynamically linked shared library ppcPython (for architecture i386): Mach-O dynamically linked shared library i386 Hmm .. i386? Oh dear, I'm running Snow Leopard and I generate 64 bit code. ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
