Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 11:23:26AM +0500, Andrey Rahmatullin wrote: > On Wed, Jun 08, 2022 at 10:43:57PM +0100, Julian Gilbey wrote: > > I'd like to ask for some help. I'm working on packaging pydevd, which > > builds a private .so library. Ordinary extensions built using cython > > or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so", > > but this library, which is not dependent on the Python version, should > > presumably be called "bar.x86_64-linux-gnu.so". > If it's just a private library and not a Python module it should be called > bar.so. > > > Question 1: How do I determine (within Python) the triplet to use when > > building the library? > You don't. > > > Question 2: How do I determine (within Python) the triplet to use when > > loading the library at runtime? > You don't, but also how are you actually loading it? Well, the upstream wanted to compile two versions of the library, one for 64 bit architectures and one for 32 bit architectures. I don't really want to build two different arch libraries in a single build, because that seems very contrary to the way the Debian architectures work, and would also limit it to the amd64/i386 architectures for no obviously good reason. I had imagined that if there is some sort of multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names. But I've never done that myself, so I've no idea if it's even meaningful to do this. The library is loaded into gdb using this code: cmd.extend([ "--eval-command='call (void*)dlopen(\"%s\", 2)'" % target_dll, "--eval-command='sharedlibrary %s'" % target_dll_name, "--eval-command='call (int)DoAttach(%s, \"%s\", %s)'" % ( is_debug, python_code, show_debug_info) ]) where cmd is a list containing the gdb call, and target_dll and target_dll_name point to the shared library. Best wishes, Julian
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 08:42:28AM +0100, Julian Gilbey wrote: > > > I'd like to ask for some help. I'm working on packaging pydevd, which > > > builds a private .so library. Ordinary extensions built using cython > > > or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so", > > > but this library, which is not dependent on the Python version, should > > > presumably be called "bar.x86_64-linux-gnu.so". > > If it's just a private library and not a Python module it should be called > > bar.so. > > > > > Question 1: How do I determine (within Python) the triplet to use when > > > building the library? > > You don't. > > > > > Question 2: How do I determine (within Python) the triplet to use when > > > loading the library at runtime? > > You don't, but also how are you actually loading it? > > Well, the upstream wanted to compile two versions of the library, one > for 64 bit architectures and one for 32 bit architectures. I don't > really want to build two different arch libraries in a single build, > because that seems very contrary to the way the Debian architectures > work, and would also limit it to the amd64/i386 architectures for no > obviously good reason. I had imagined that if there is some sort of > multiarch setup, one might have the amd64 and i386 packages installed > simultaneously, hence the need for different names. The normal way for this is putting it into /usr/lib//pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time. You can get the triplet in d/rules, not sure how should you pass it to the build system as that depends on the build system used. For the run time, https://wiki.debian.org/Python/MultiArch suggests sys.implementation._multiarch (note that you cannot use it during the build as that would break cross-compilation etc.), not sure if there are better ways. -- WBR, wRAR signature.asc Description: PGP signature
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 01:03:25PM +0500, Andrey Rahmatullin wrote:
> [...]
> > Well, the upstream wanted to compile two versions of the library, one
> > for 64 bit architectures and one for 32 bit architectures. I don't
> > really want to build two different arch libraries in a single build,
> > because that seems very contrary to the way the Debian architectures
> > work, and would also limit it to the amd64/i386 architectures for no
> > obviously good reason. I had imagined that if there is some sort of
> > multiarch setup, one might have the amd64 and i386 packages installed
> > simultaneously, hence the need for different names.
> The normal way for this is putting it into
> /usr/lib//pkgname/foo.so, and according to the code below you'll
> need the full path at the run time so you indeed need the triplet at both
> build and run time. You can get the triplet in d/rules, not sure how
> should you pass it to the build system as that depends on the build system
> used. For the run time, https://wiki.debian.org/Python/MultiArch suggests
> sys.implementation._multiarch (note that you cannot use it during the
> build as that would break cross-compilation etc.), not sure if there are
> better ways.
Thanks for your help!
OK (and yes, it does require the full path at runtime). What triplet
do I use in d/rules? dpkg-architecture offers 6 different ones:
DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing
DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
to confirm that.
About the location, though: why do compiled Python libraries live in
/usr/lib/python3/dist-packages/ and not
/usr/lib//? And is there a good reason not to do
the same with this Python-package-specific library? It's not for
general use, so I can't see why I shouldn't put it in the python3
directory with the other compiled Python module libraries.
Best wishes,
Julian
Re: Build and run-time triplets
On Thu, 09 Jun 2022 at 13:03:25 +0500, Andrey Rahmatullin wrote:
> The normal way for this is putting it into
> /usr/lib//pkgname/foo.so, and according to the code below you'll
> need the full path at the run time so you indeed need the triplet at both
> build and run time.
You can do something like
handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);
(that's a literal string passed to dlopen!) and ld.so/libdl will expand
${LIB} at runtime to the token that was configured into our glibc, which
in Debian's case is "lib/x86_64-linux-gnu" or similar. On non-Debian
distributions, ${LIB} typically expands to "lib64" or "lib32" or "lib"
instead, whichever one is most appropriate for the architecture and the
distro's layout.
Then you'd install the private library into what Autotools would refer to
as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems)
and it will usually end up in the correct place. This assumes that
${libdir} is configured to something like
${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate
for the distribution, but that's normally true anyway, and in particular
should be true in debhelper.
Replace /usr with the ${exec_prefix} determined at compile-time if you
want to send code upstream that supports custom installation prefixes.
(Historical note: this is broken on ancient/EOL versions of Debian and
maybe some of the Extended Security Maintenance versions of Ubuntu,
where ${LIB} mistakenly expanded to the multiarch tuple without the
"lib/" prefix. Non-FHS distributions like Exherbo and NixOS might also
need to adjust it a bit, but they have to adjust everything else anyway,
so they should be used to that...)
smcv
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 10:26:13AM +0100, Simon McVittie wrote:
> On Thu, 09 Jun 2022 at 13:03:25 +0500, Andrey Rahmatullin wrote:
> > The normal way for this is putting it into
> > /usr/lib//pkgname/foo.so, and according to the code below you'll
> > need the full path at the run time so you indeed need the triplet at both
> > build and run time.
>
> You can do something like
>
> handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);
> [...]
>
> Then you'd install the private library into what Autotools would refer to
> as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems)
> and it will usually end up in the correct place. This assumes that
> ${libdir} is configured to something like
> ${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate
> for the distribution, but that's normally true anyway, and in particular
> should be true in debhelper.
Thanks Simon!
The build system here is the standard Python setup.py, except for this
library. That is built by the following script:
---
g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
mv attach_linux_amd64.so ../attach_linux_amd64.so
echo Compiled amd64
---
There's not even an attempt at working out ${libdir} or so on. It
seems like overkill to set up a whole Autotools environment for this
one file :-(
I'm still unsure why I shouldn't just put it in the Python package
area near the cython files such as
/usr/lib/python3/dist-packages/_pydevd_bundle/pydevd_cython.cpython-310-x86_64-linux-gnu.so
Best wishes,
Julian
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 09:56:42AM +0100, Julian Gilbey wrote:
> > [...]
> > > Well, the upstream wanted to compile two versions of the library, one
> > > for 64 bit architectures and one for 32 bit architectures. I don't
> > > really want to build two different arch libraries in a single build,
> > > because that seems very contrary to the way the Debian architectures
> > > work, and would also limit it to the amd64/i386 architectures for no
> > > obviously good reason. I had imagined that if there is some sort of
> > > multiarch setup, one might have the amd64 and i386 packages installed
> > > simultaneously, hence the need for different names.
> > The normal way for this is putting it into
> > /usr/lib//pkgname/foo.so, and according to the code below you'll
> > need the full path at the run time so you indeed need the triplet at both
> > build and run time. You can get the triplet in d/rules, not sure how
> > should you pass it to the build system as that depends on the build system
> > used. For the run time, https://wiki.debian.org/Python/MultiArch suggests
> > sys.implementation._multiarch (note that you cannot use it during the
> > build as that would break cross-compilation etc.), not sure if there are
> > better ways.
>
> Thanks for your help!
>
> OK (and yes, it does require the full path at runtime). What triplet
> do I use in d/rules? dpkg-architecture offers 6 different ones:
> DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing
> DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
> to confirm that.
It's always DEB_HOST_MULTIARCH. *_TARGET_* are for Canadian cross. See
TERMS in dpkg-architecture(1).
> About the location, though: why do compiled Python libraries live in
> /usr/lib/python3/dist-packages/ and not
> /usr/lib//?
That's where the import machinery expects them. They also have special
names.
> And is there a good reason not to do the same with this
> Python-package-specific library?
Yes, it's not Python-related.
> It's not for general use, so I can't see why I shouldn't put it in the
> python3 directory with the other compiled Python module libraries.
It's not a module.
--
WBR, wRAR
signature.asc
Description: PGP signature
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 10:43:01AM +0100, Julian Gilbey wrote:
> > > The normal way for this is putting it into
> > > /usr/lib//pkgname/foo.so, and according to the code below you'll
> > > need the full path at the run time so you indeed need the triplet at both
> > > build and run time.
> >
> > You can do something like
> >
> > handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);
> > [...]
> >
> > Then you'd install the private library into what Autotools would refer to
> > as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems)
> > and it will usually end up in the correct place. This assumes that
> > ${libdir} is configured to something like
> > ${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate
> > for the distribution, but that's normally true anyway, and in particular
> > should be true in debhelper.
>
> Thanks Simon!
>
> The build system here is the standard Python setup.py, except for this
> library. That is built by the following script:
>
> ---
> g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
> mv attach_linux_amd64.so ../attach_linux_amd64.so
> echo Compiled amd64
> ---
>
> There's not even an attempt at working out ${libdir} or so on.
Sure, it's not necessary to know the installation path at the compile and
link time. It's only needed at the install time.
--
WBR, wRAR
signature.asc
Description: PGP signature
Re: Build and run-time triplets
On Thu, 09 Jun 2022 at 09:56:42 +0100, Julian Gilbey wrote:
> OK (and yes, it does require the full path at runtime). What triplet
> do I use in d/rules? dpkg-architecture offers 6 different ones:
> DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing
> DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
> to confirm that.
You'd want DEB_HOST_MULTIARCH here (or use ${LIB} as I mentioned in a
previous message to this thread).
DEB_BUILD_* is the architecture you are building *on*. For example,
if you cross-build ARM packages on an x86 machine, DEB_BUILD_* is x86.
Only use this rarely, for instance if you are compiling a program that
you will run during the build but not install into the package. If you
don't know that this is the correct choice then it probably isn't.
DEB_HOST_* is the architecture you will run the built code on. For example,
if you cross-build ARM packages on an x86 machine, DEB_HOST_* is ARM.
This is usually the interesting one in practice.
DEB_TARGET_* is almost never relevant: it only matters if you are
compiling a compiler.
*_MULTIARCH is the normalized multiarch tuple, for example i386-linux-gnu
on the i386 architecture.
*_GNU_TYPE is the GNU tuple to be used by Autotools build systems and
to choose a cross-compiler, for example i686-linux-gnu on the i386
architecture.
> About the location, though: why do compiled Python libraries live in
> /usr/lib/python3/dist-packages/ and not
> /usr/lib//?
The Python jargon for a native C/C++ library that can be loaded to
provide a Python module is an *extension*.
If a program will load a shared object as a plugin, then the shared object
needs to be installed into whatever directory the program has chosen as
the location where it looks for plugins, otherwise the program will not
find it: it is the program's choice, not the plugin's choice.
Python extensions are like plugins for Python, and Python has chosen
/usr/lib/python3/dist-packages as one of several locations where it looks
for extensions (specifically, extensions that come from a package other
than Python itself), so that is where you have to put them. If the extension
that implements the "foo" module is somewhere else, then Python code that
does "import foo" won't find it.
(Analogous: if a package wants to provide a plugin for GStreamer, it has
to put it in /usr/lib//gstreamer-1.0. If it puts the plugin
anywhere else, GStreamer will not find it and the plugin will be useless.)
The shared object also needs to follow the naming convention and API that
the program that will load it has chosen. The naming convention that Python
has chosen is that if you will load the module with "import foo", then
the filename is something like "foo.cpython-310-x86_64-linux-gnu.so" or
"foo.abi3.so", and it must export a symbol named "PyInit_foo" with a
particular signature.
Something that might be confusing you is that for a lot of Python modules
that are implemented in C, the thing that API users are meant to import
directly is Python code, and the Python code internally imports an
extension with a different name. For instance, to use python3-dbus,
you "import dbus", but the lower-level C code that implements that is
actually in a private extension named "_dbus_bindings" (so the filename is
_dbus_bindings.cpython-310-x86_64-linux-gnu.so). As an API user, you are
not meant to "import _dbus_bindings", but the Python code in python3-dbus
needs to be able to do that in order to do its job.
> And is there a good reason not to do
> the same with this Python-package-specific library?
If it isn't a Python extension (cannot be loaded into Python with
"import foo" to provide a module API to Python code) then it would seem
inappropriate to put it in the directory that is reserved for Python
extensions.
> It's not for
> general use, so I can't see why I shouldn't put it in the python3
> directory with the other compiled Python module libraries.
Because it isn't a Python extension (what you are calling a "compiled
Python module library"), but instead some other sort of library with a
different purpose (from your other messages, it seems like it's being
injected into a program that is being debugged with gdb).
smcv
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 11:00:28AM +0100, Simon McVittie wrote:
> On Thu, 09 Jun 2022 at 09:56:42 +0100, Julian Gilbey wrote:
> > OK (and yes, it does require the full path at runtime). What triplet
> > do I use in d/rules? dpkg-architecture offers 6 different ones:
> > DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing
> > DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
> > to confirm that.
>
> You'd want DEB_HOST_MULTIARCH here (or use ${LIB} as I mentioned in a
> previous message to this thread).
> [...]
Thanks for this great explanation - I'm learning so much in this
thread!
> > About the location, though: why do compiled Python libraries live in
> > /usr/lib/python3/dist-packages/ and not
> > /usr/lib//?
>
> The Python jargon for a native C/C++ library that can be loaded to
> provide a Python module is an *extension*.
> [...]
> > And is there a good reason not to do
> > the same with this Python-package-specific library?
>
> If it isn't a Python extension (cannot be loaded into Python with
> "import foo" to provide a module API to Python code) then it would seem
> inappropriate to put it in the directory that is reserved for Python
> extensions.
I hear this argument, though we do find data files and other sorts of
things other than Python files and compiled extensions that are needed
by Python modules in that same directory ("data_files" in setup.py);
for example numpy has dozens of other files in
/usr/lib/python3/dist-packages/numpy.
I think the counter-argument is exactly what you said: as long as the
shared library is in a place where the code that uses it knows where
to locate it, all will work fine. The Python code as-written expects
the library to be in the same directory as the Python code.
Best wishes,
Julian
Re: Build and run-time triplets
On Thu, Jun 09, 2022 at 03:00:24PM +0500, Andrey Rahmatullin wrote:
> > The build system here is the standard Python setup.py, except for this
> > library. That is built by the following script:
> >
> > ---
> > g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
> > mv attach_linux_amd64.so ../attach_linux_amd64.so
> > echo Compiled amd64
> > ---
> >
> > There's not even an attempt at working out ${libdir} or so on.
> Sure, it's not necessary to know the installation path at the compile and
> link time. It's only needed at the install time.
Ah, to clarify: this script compiles the library and sticks it into
the parent directory. It is then installed with the rest of the
Python module via setup.py into /usr/lib/python3/dist-packages/...
So if I want to install it somewhere else, I will have to handle that
manually.
Best wishes,
Julian
