Dear George
On 08/10/2019 11:21, George Sheldrick wrote:
As explained in the attached email from Peter Keller, I was
deliberately preparing the binary Linux SHELX distribution using older
(2011) system libraries so that the programs would run on older
systems that many users are still using. Unfortunately this means that
they do not run on some recent cutting edge distributions including
Debian 10. This can be fixed with 'vsyscall=emulate' when installing
the OS but not all users may be allowed to do this. Dynamic binaries
would be smaller but would require the user to provide the right
libraries.
If I prepare statically linked binaries (using the latest ifort and
ubuntu) they appear to run on current systems but may have problems on
older systems. I may have to offer (e.g. in CCP4 and on the SHELX
server) two sets of Linux binaries in the future, one for 'vintage'
systems and one for current systems.
This is treating the two possibilities as different architectures. This
is a viable approach. Having said that, I doubt that anyone is using a
system now that only supports the vsyscall method for time() etc. I
suspect that linking with a suitable version of glibc would produce a
portable, static executable.
Then, there is also the package approach that David mentioned, although
that requires learning new tools.
Alternatively I could provide both statically and dynamically linked
versions.
There are two approaches for rolling your own dynamically-linked
executables that avoid the need for users to supply the required
libraries themselves:
(1) Link most libraries statically, using link options like
'-Wl,-Bstatic,-lsomelib1,-lsomelib2,-Bdynamic'. This is the kind of
approach that Clemens Vonrhein is using for a future release of Global
Phasing software. The final executables only have minimal run-time
dependencies:
% ldd `which process`
linux-vdso.so.1 (0x00007ffffb54f000)
libc.so.6 => /lib64/libc.so.6 (0x00007f144842a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f14487e4000)
If a system cannot provide these dependencies, the sysadmin will have
bigger problems to solve than SHELXD not working ;-)
(2) Ship any required libraries that are not provided by the system
along with your binaries. This is the approach that CCP4 is taking:
% ldd `which mtzdump`
linux-vdso.so.1 (0x00007ffe039e1000)
libccp4f.so.0 =>
/public/xtal/ccp4/7.0/official/Linux-x86_64/ccp4-7.0/bin/../lib/libccp4f.so.0
(0x00007fbdd53f5000)
libccp4c.so.0 =>
/public/xtal/ccp4/7.0/official/Linux-x86_64/ccp4-7.0/bin/../lib/libccp4c.so.0
(0x00007fbdd51bd000)
libgfortran.so.3 =>
/public/xtal/ccp4/7.0/official/Linux-x86_64/ccp4-7.0/bin/../lib/libgfortran.so.3
(0x00007fbdd4ecb000)
libm.so.6 => /lib64/libm.so.6 (0x00007fbdd4b93000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbdd497b000)
libc.so.6 => /lib64/libc.so.6 (0x00007fbdd45c1000)
libmmdb2.so.0 =>
/public/xtal/ccp4/7.0/official/Linux-x86_64/ccp4-7.0/bin/../lib/../lib/libmmdb2.so.0
(0x00007fbdd42af000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbdd7f5e000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fbdd3f25000)
The path to the directory containing CCP4-supplied shared libraries is
encoded in the executable itself:
% readelf -d `which mtzdump`
Dynamic section at offset 0x7650 contains 31 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libccp4f.so.0]
0x0000000000000001 (NEEDED) Shared library: [libccp4c.so.0]
0x0000000000000001 (NEEDED) Shared library:
[libgfortran.so.3]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib]
Note the special token "$ORIGIN", which means the directory containing
the executable (this is documented in the 'man' page for ld.so). The
RPATH item is set to the above value with an option like
'-Wl,-rpath,\$ORIGIN/../lib' (where the '$' character needs to be
escaped from the shell, obviously).
Note that -rpath is not the same as -L: -L is used at link-time to find
the libraries needed to check that all the symbols are resolvable.
-rpath is used to locate shared libraries at run-time: its argument
doesn't even have to exist at link-time. If you go down this route, you
will need to specify both -L and -rpath.
What do users think?
As for which is the best approach, I'll let others make their opinions
known, but I hope that this explanation has helped.
Regards,
Peter.
Best wishes, George
On 08.10.19 11:04, Peter Keller wrote:
HI Tim,
On Mon, 7 Oct 2019, Tim Gruene wrote:
Date: Mon, 07 Oct 2019 23:04:28 +0200
From: Tim Gruene <tim.gru...@univie.ac.at>
To: Peter Keller <pkel...@globalphasing.com>
Cc: CCP4BB@jiscmail.ac.uk
Subject: Re: [ccp4bb] Shelx and debian 10
@Peter: are you sure that without 'vsyscall=emulate' linux binaries
need to be
dynamically linked? I would be very surprised if the linux kernel would
disable statically linked binaries. I rather think that the vanilla
versions
of shelx c/d/e (from shelx.uni-goettingen.de) are compiled with an
obsolete
compiler / obsolete compiler options.
You're right, I wrote my reply to Bernhard too rapidly, and conflated
two separate issues. Debian 10 still supports static binaries of
course, but in its default configuration (without vsyscall=emulate),
those static binaries must be linked with a version of glibc that
doesn't require vsyscalls. OTOH, dynamic binaries don't suffer from
this problem, because they can use vDSO provided by the running
(rather than the build) system.
I think that part of the problem is that traditionally when we want
to build portable linux binaries, we tend to build on the oldest
distribution that we want to support, relying on backwards
compatibility to provide the portability that we are after. We often
build statically, because it is a robust way of including all the
required libraries and is less fiddly and error-prone than providing
them as dynamic libraries. There is also no danger of breakage caused
by rogue values of LD_LIBRARY_PATH (which users shouldn't be setting
of course, but we have no way of stopping them). The drawback of this
approach is that when backwards compatibility is broken, there is no
application-level fix.
These kinds of problems are rare, but when they do happen the onus is
on those of us who distribute binary applications to find solutions.
Some sysadmins may have good reasons for being reluctant to change
kernel parameters to get third-party applications to work.
Regards,
Peter.
On Monday, October 7, 2019 5:53:44 PM CEST Peter Keller wrote:
Dear Bernhard,
We had this issue drawn to our attention last year by an early adopter
of Debian 10 while it was still in testing. I thought that it was a
bug,
and submitted a report accordingly here:
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=889965>. I was told
that it is not a bug, but a feature ;-)
If you are able, you could try setting the kernel parameter
vsyscall=emulate. In the longer term, SHELXC/D/E will have to be
rebuilt
to support systems where the vsyscall has been disabled. This means
they
have to be dynamic executables that include the following in the
output
of 'ldd':
% ldd /bin/bash
linux-vdso.so.1 (0x00007fff50952000)
....
All current distros use vDSO, so this shouldn't cause portability
problems by itself, but handling dynamic executables can be trickier
than static ones.
For a little more background, see <https://lwn.net/Articles/446528/>
Finally, you have my commiserations: although this change has been a
long time coming, it hasn't attracted a lot of attention. It was bound
to catch users of static executables by surprise.
Regards,
Peter.
On 07/10/2019 16:05, Bernhard Rupp wrote:
Hi Fellows,
we updated to Debian 10 on the local workshop computers, and
reinstalled
Coot and ccp4. All fine.
Problem: Shelxc/d/e/ does not run, and
the call exits immediately sans any message.
This holds for the binaries included in ccp4 as well as for those
from
the SHELX site.
The executables from CCP4 and SHELX site – same file size, probably
same - run fine under Debian 9.
I suspect a library problem.
Does some kind soul have CDE binaries for Debian 10 to share?
Many thx in advance, BR
--------------------------------------------------------------------------
--------------
Bernhard Rupp
Department of Genetics and Pharmacology
Institute of Genetic Epidemiology
Medical University Innsbruck
Schöpfstrasse 41
A 6020 Innsbruck – Austria
+43 (676) 571-0536
bernhard.r...@i-med.ac.at
--------------------------------------------------------------------------
--------------
k.k. Hofkristallamt
San Diego, CA 92084
001 (925) 209-7429
b...@ruppweb.org
b...@hofkristallamt.org
http://www.ruppweb.org/
-----------------------------------------------------------------------
------------------------------------------------------------------------
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
