Re: [R-pkg-devel] Best practices for built version checking in packages LinkingTo others?
Dear All, I think I have something that works and may even work as a more general API for checking for LinkingTo ABI compatibility. It's in 'ergm' 4.8.1 on CRAN, but would it be useful to anyone else if I posted an explanation of how it works somewhere? Would a package that provides this API for saving and checking ABI versions be useful to anyone? Best, Pavel On Sun, 2025-01-19 at 05:31 +, Pavel Krivitsky via R-package-devel wrote: > Thanks, Ivan! Replies below. > > On Fri, 2025-01-17 at 16:50 +0300, Ivan Krylov wrote: > > В Fri, 17 Jan 2025 10:21:27 +0000 > > Pavel Krivitsky via R-package-devel > > пишет: > > > > > 1. What is the best way to save this information? I have a > > > rudimentary implementation [1] > > > > For now, this is the best way we have. Export the ABI version as an > > include-time constant and as a registered callable. The LinkingTo > > reverse dependencies can give the include-time constant to the > > registered callable to compare the two versions. > > Ben Bolker pointed out that since the code in R/ is executed at > build- > time, it is also possible to store the build-time ABI information > entirely in R; the downside of this approach is that it requires > instrumenting every client package rather than just the library > package. > > > I agree that we could do better and introduce an ABIversion: field > > to > > the DESCRIPTION file. R CMD INSTALL would write down the current > > ABIversion of every package that the current package is LinkingTo > > (into the installed package's DESCRIPTION or Meta/features.rds). > > Later, the namespace loader would check the ABI versions and > > provide > > an informative message in case a LinkingTo dependency is updated > > with > > an incremented ABI version. A machine-readable list of ABI-level > > dependencies should simplify the lives of the binary package > > maintainers as well. > > Yes, that would be great to have. (One nitpick: not just incremented, > but different in either direction.) > > AFAIK, for those platforms that use binaries, there is already a > mechanism for downloading a different ZIP for a different version of > R, > but I believe it's accomplished simply by having a directory for > every > version of R with a potentially different ABI. > > I suppose this could be generalised along the lines of a directory > hierarchy of the form > > /bin/windows/contrib/RVERSION/PKG_ABIVERSION/.../PKG_ABIVERSION/PKG_P > KGVERSION.zip > > with a nesting level for every LinkingTo package, in lexicographic > order. Then, given the installed R version and the installed versions > of each package in the LinkingTo list, a unique URL can be > constructed. > However, I don't know how practical this is and how likely to be > implemented in the foreseeable future. > > > > 2. How to best test the C API without making updates > > > impossible? > > > > Would abi-compliance-checker [2] have helped there? > > I meant specifically in the context of R CMD check; the only want to > fully test the C API and associated facilities is to actually install > the client package and see if it works. > > That having been said, 'ergm' may be unusual in this, since the API > is > mostly used as a way for others to implement calculation of graph > statistics that are then operated on by functions in 'ergm'. That is, > rather than simply offering C functions for the client packages to > use, > 'ergm' calls C functions from those packages that it locates via > R_FindSymbol(). Thus, it is helpful to test it as a part of 'ergm' > itself rather than just reverse-dependency checks. > > > > 3. What should be done in case of a mismatch? > > > > It's hard to guarantee anything when the ABI is broken. Maybe it's > > completely harmless because your code will know that the new > > structure member is not initialised. Maybe the wrong stack layout > > will crash the process due to a return address mismatch, dooming > > the > > process from the first function call.k > > > > > 1. Is there a way for 'ergm' to detect when a package > > > LinkingTo it is being loaded? > > > > With inline functions, you could make your every API call perform > > ABI > > version checking, but that obviously comes with potential > > performance > > problems: > > The reason I ask is that the best time to inform the user of an ABI > change is when the client packa
Re: [R-pkg-devel] Best practices for built version checking in packages LinkingTo others?
Thanks, Ivan! Replies below. On Fri, 2025-01-17 at 16:50 +0300, Ivan Krylov wrote: > В Fri, 17 Jan 2025 10:21:27 + > Pavel Krivitsky via R-package-devel > пишет: > > > 1. What is the best way to save this information? I have a > > rudimentary implementation [1] > > For now, this is the best way we have. Export the ABI version as an > include-time constant and as a registered callable. The LinkingTo > reverse dependencies can give the include-time constant to the > registered callable to compare the two versions. Ben Bolker pointed out that since the code in R/ is executed at build- time, it is also possible to store the build-time ABI information entirely in R; the downside of this approach is that it requires instrumenting every client package rather than just the library package. > I agree that we could do better and introduce an ABIversion: field to > the DESCRIPTION file. R CMD INSTALL would write down the current > ABIversion of every package that the current package is LinkingTo > (into the installed package's DESCRIPTION or Meta/features.rds). > Later, the namespace loader would check the ABI versions and provide > an informative message in case a LinkingTo dependency is updated with > an incremented ABI version. A machine-readable list of ABI-level > dependencies should simplify the lives of the binary package > maintainers as well. Yes, that would be great to have. (One nitpick: not just incremented, but different in either direction.) AFAIK, for those platforms that use binaries, there is already a mechanism for downloading a different ZIP for a different version of R, but I believe it's accomplished simply by having a directory for every version of R with a potentially different ABI. I suppose this could be generalised along the lines of a directory hierarchy of the form /bin/windows/contrib/RVERSION/PKG_ABIVERSION/.../PKG_ABIVERSION/PKG_PKGVERSION.zip with a nesting level for every LinkingTo package, in lexicographic order. Then, given the installed R version and the installed versions of each package in the LinkingTo list, a unique URL can be constructed. However, I don't know how practical this is and how likely to be implemented in the foreseeable future. > > 2. How to best test the C API without making updates impossible? > > Would abi-compliance-checker [2] have helped there? I meant specifically in the context of R CMD check; the only want to fully test the C API and associated facilities is to actually install the client package and see if it works. That having been said, 'ergm' may be unusual in this, since the API is mostly used as a way for others to implement calculation of graph statistics that are then operated on by functions in 'ergm'. That is, rather than simply offering C functions for the client packages to use, 'ergm' calls C functions from those packages that it locates via R_FindSymbol(). Thus, it is helpful to test it as a part of 'ergm' itself rather than just reverse-dependency checks. > > 3. What should be done in case of a mismatch? > > It's hard to guarantee anything when the ABI is broken. Maybe it's > completely harmless because your code will know that the new > structure member is not initialised. Maybe the wrong stack layout > will crash the process due to a return address mismatch, dooming the > process from the first function call.k > > > 1. Is there a way for 'ergm' to detect when a package > > LinkingTo it is being loaded? > > With inline functions, you could make your every API call perform ABI > version checking, but that obviously comes with potential performance > problems: The reason I ask is that the best time to inform the user of an ABI change is when the client package is loaded, and it would be helpful if this could be done without modifying the client package. Is there a mechanism for the library package to monitor packages being loaded, and for each one check if it's LinkingTo it to do the version check? > > 2. What should happen if a mismatch is detected? Should > > the > > loading be blocked (e.g., by having .onLoad() throw an error), or > > should it lead to a "use at your own risk" type warning? > > The safest option would be to signal an error before any of the > potentially incompatible functions are called. It's probably > pragmatic to provide an "I know what I'm doing, void my warranty" > emergency override. Probably an options() setting. Best, Pavel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] Best practices for built version checking in packages LinkingTo others?
Dear All,
I maintain a package ('ergm') that exports a C API via /inst/include/
that includes header files with data structures, macros, and inline
functions, as well as a number of functions exported in a stubs file
(that use R_FindSymbol() and/or R_GetCCallable() to locate the
functions in 'ergm') that the linking package #includes to access them.
I've recently had an update stall because a source-compatible change in
one of the structs changed the ABI---that is, packages LinkingTo 'ergm'
needs to be rebuilt, and one of them was Suggested by 'ergm' precisely
in order to test the exported C API. As
https://cloud.r-project.org/doc/manuals/r-devel/R-exts.html#Linking-to-native-routines-in-other-packages
suggests, LinkingTo packages should either depend on an exact version
(which is impractical) or test whether the version matches at runtime.
Thus, the questions are:
1. What is the best way to save this information? I have a
rudimentary implementation [1], but R already has a mechanism to
store the build-time R version since it warns about that; can it
be used for packages as well?
2. How to best test the C API without making updates impossible?
Obviously, I have my own continuous integration setups, so I
could simply disable the tests on CRAN, but there is some benefit
to having tests on CRAN as well [2]. That is, should the testing
be conditional ABI versions not mismatching, or is this more
trouble than it's worth?
3. What should be done in case of a mismatch?
1. Is there a way for 'ergm' to detect when a package LinkingTo it
is being loaded? Checking and handling can be put into .onLoad()
of the LinkingTo packages, but there is some benefit to putting
as little code into the LinkingTo package. However, as far as I
can tell, hooks can only be set for specific packages. Is this
correct?
2. What should happen if a mismatch is detected? Should the loading
be blocked (e.g., by having .onLoad() throw an error), or should
it lead to a "use at your own risk" type warning?
Thanks in advance,
Pavel
[1]
This can be implemented by defining macros with the ABI version in
'ergm', e.g.,
ergm_version.h:
#define ERGM_API_MAJOR 4
#define ERGM_API_MINOR 8
typedef struct {unsigned int major, minor;} APIVersion;
then
ergm_stubs.c (#included from every LinkingTo package into a C file):
#include "ergm_version.h"
APIVersion GetBuiltErgmAPIVersion(void){
return (APIVersion) {ERGM_API_MAJOR, ERGM_API_MINOR};
}
Then, at compile time, the macros will be resolved so
GetBuiltErgmAPIVersion() will return the build-time 'ergm' version.
R_FindSymbol() could be used to obtain a pointer to
GetBuiltErgmAPIVersion() from a given package and hence its 'ergm'
build version, which can then be returned to R.
[2]
For example (and this actually happened several versions ago) the
package might do something with computational cost that depends on the
number of installed packages.
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
