On Tue, Apr 22, 2025 at 8:53 AM Marc Culler <marc.cul...@gmail.com> wrote:
> Here is one thing that I find strange. The only external libraries that > are needed to compile python on macOS are openssl and (if you want the real > gnu thing) readline. I know, because I have compiled python on macOS. The > dependencies like libbz2, liblzma, libffi, libz and even libsqlite3 are all > provided with the OS. They are available on any macOS system and > accessible by any xcode compiler. Yet Sage does not find "usable system > versions" of many of them. (It does find bzip2, for some weird reason, and > maybe the liblzma behavior is explained by the fact that Apple does not > provide an xz executable even though it does provide liblzma.) > > The following are available as part of the OS, for example: > libbz2, libblas, libdbm, libffi, libf77lapack, liblapack, libiconv, > liblzma, libncurses, libsqlite3, libz (many versions) > > The point, to be clear, is that Sage will currently not accept the > versions of these libraries provided by Apple, but python will accept the > subset which it needs. Yet somehow the claim is being made that the > "remove the python spkg" project will make it be the case that sage will > accept a python built locally even though, if done in the obvious way, such > a python build will use many of those Apple libraries. At the same time > there is a worry about the crashes that will occur if python uses one > version of a library and some sage package uses a different one. Yet there > is no option to use the Apple libraries to satisfy the requirements of Sage > packages which use versions of those libraries. > That's the first time I see a complaint that the Apple's libraries used for Python or its modules are not accepted. This is something we can fix, to an extent. And surely we can adjust the check for xz. Regarding macOS native bzip2, it just happens to pass our tests, nothing weird about it. Regarding libz, it, implictly, needs pkg-config. One can hand-craft pkg-config .pc file for it, then macOS native libz probably will work, I didn't try. Regarding Apple's lapack/blas, i.e. Accelerate Framework libraries, there are upstream packages which use openblas on macOS - the most prominent such package is scipy. scipy maintainers dropped Accelerate Framework as it was lagging many lapack/blas versions behind compared to the lapack/blas reference implementation (and openblas). (there were more bugs IIRC). I am not sure we want different Python modules using different lapack/blas implementations. Dima > > These are all reasons why the current python spkg exists. And this is > part of why I think it is inevitable that the end of the "remove the python > spkg" story will be that the only supported external pythons on macOS will > be homebrew and conda. I am not allergic to homebrew, but I don't think > anyone other than me is going to pay any attention to the issues of how to > make the sage distribution self-contained, relocatable and compatible with > a wide range of macOS versions when it is built against a homebrew > distribution. Despite all of the glib pronouncements about how trivial > this must be, I don't think it will be a trivial thing to do. On the other > hand it is obvious, from the fact that an idiot like me was able to do it, > that creating a self-contained relocatable Sage distribution with the > current python spkg configuration *is* trivial. > > - Marc > On Monday, April 21, 2025 at 11:35:08 PM UTC-5 dim...@gmail.com wrote: > >> On Mon, Apr 21, 2025 at 9:41 PM Marc Culler <marc....@gmail.com> wrote: >> >>> Thank you Michael, >>> >>> My first guess is that >>> SAGE_SPKG_DEPCHECK([bzip2 liblzma libffi zlib], ... >>> >>> >>> I think that your guess is correct. >>> >>> IIRC you are building everything (except python, now) from SPKGs. If >>> so and if you are sure that your newly-built python was linked against >>> the sage copies of bzip2, zlib, etc., you can just ignore the DEPCHECK >>> (edit it out). >>> >>> >>> Except that it is impossible for the newly-built python to be linked >>> against the >>> sage copies of those libs, because the hewly-built python needs to be >>> built >>> *before* sage is built, and therefore before those libs have been built. >>> >>> I assume that what you really mean is that a newly-built python is not >>> sufficient. >>> That python must be accompanied by all of those other libraries and >>> then sage >>> must also use the same libraries. One way to do that might be to modify >>> the sage >>> build process so that it builds the spkgs for those libraries and then >>> is forced to >>> halt while a separate python compilation is done in such a way that the >>> python >>> modules which use those libraries can be linked against the versions >>> which were >>> built by the corresponding sage spkgs. The sage build process could >>> perhaps then >>> be restarted to build the rest of sage. >>> >>> Of course that kind of craziness is exactly what I am hoping to avoid >>> and what I do >>> not have to worry about with the current setup. All of that complicated >>> sequencing >>> is currently handled automatically because python3 is built by an spkg >>> which specifies >>> those libraries as dependencies. >>> >>> So the only practical way forward seems to be to install python in some >>> context which >>> also includes all of those other libraries. And somehow sage must be >>> informed to >>> search in that context for all of those libraries. (Running configure >>> --help shows >>> options for saying whether to use the "system versions" of those >>> libraries, but there >>> seems to be no way of providing a path to those options, and there is no >>> hint about >>> how sage will actually decide whether the "system version" of one of >>> those libraries >>> is usable.) >>> >>> I expect that decision process isn't something general, such as checking >>> for the >>> libraries in the same directory where the python3.13 directory is >>> located, or in the lib >>> directory adjacent to the bin directory containing the python3 >>> executable, or a hardwired >>> test which causes sage to use the versions of those libraries provided >>> by Apple,. I would >>> guess that instead there are some hardwired checks for whether homebrew >>> or conda >>> is being used, and maybe a check of /usr/local. >>> >> >> Not really. You should be able to get away with appropriate settings of >> PATH, CFLAGS and LDFLAGS (to point at >> appropriate locations of the executables, headers, and dylibs. >> >> >>> If those checks fail I would guess >>> that it just returns an error. Does anyone know how sage decides >>> whether there >>> is a usable "system version" of bzip2 or liblzma? >>> >> >> $ cat build/pkgs/bzip2/spkg-configure.m4 >> SAGE_SPKG_CONFIGURE([bzip2], [ >> AC_CHECK_HEADER(bzlib.h, [ >> AC_SEARCH_LIBS([BZ2_bzCompress], [bz2], [ >> AC_PATH_PROG([bzip2_prog], [bzip2]) >> AS_IF([test x$bzip2_prog = x], [sage_spkg_install_bzip2=yes]) >> ], [sage_spkg_install_bzip2=yes]) >> ], [sage_spkg_install_bzip2=yes]) >> ]) >> That is, for bzip2, all you need is that your compiler knows where >> bzlib.h is, that your linker knows where to find libbz2 >> (which should have BZ2_bzCompress function) >> and that bzip2 executable is in your PATH >> >> $ cat build/pkgs/liblzma/spkg-configure.m4 >> SAGE_SPKG_CONFIGURE([liblzma], [ >> SAGE_SPKG_DEPCHECK([xz], [ >> dnl The library is actually installed by the xz spkg. >> AC_CHECK_LIB([lzma], [lzma_raw_decoder], [lzma_cv_liblzma=yes], >> [lzma_cv_liblzma=no]) >> AC_CHECK_HEADER([lzma.h], [lzma_cv_lzma_h=yes], [lzma_cv_lzma_h=no]) >> if test "$lzma_cv_liblzma" != "yes" -o "$lzma_cv_lzma_h" != "yes"; >> then >> sage_spkg_install_liblzma=yes >> fi >> ]) >> ]) >> >> That is, for liblzma first it's checked that xz from the system is >> available; then, it's checked that your compiler >> finds lzma.h, and your linker finds liblzma (with lzma_raw_decoder >> function). >> >> speaking of xz, you need xz in your PATH. However, >> please note the comments at the bottom of the following (that is, one >> doesn't really need xz) >> >> $ cat build/pkgs/xz/spkg-configure.m4 >> SAGE_SPKG_CONFIGURE([xz], [ >> AC_CACHE_CHECK([for xz >= 4.999.0], [ac_cv_path_XZ], [ >> AC_PATH_PROGS_FEATURE_CHECK([XZ], [xz], [ >> xz_version=`$ac_path_XZ --version 2>&1 | cut -d' ' -f4 | >> $SED -n 1p` >> AS_IF([test -n "$xz_version"], [ >> AX_COMPARE_VERSION([$xz_version], [ge], [4.999.0], [ >> ac_cv_path_XZ="$ac_path_XZ" >> ac_path_XZ_found=: >> ]) >> ]) >> ]) >> AS_IF([test -z "$ac_cv_path_XZ"], [sage_spkg_install_xz=yes]) >> ]) >> ], [dnl REQUIRED-CHECK >> dnl Issue #30948: All dependencies on "xz" are merely build-time >> dependencies >> dnl on the xz binary (for unpacking the tarball in >> sage_bootstrap.uncompress.tar_file >> dnl - when sage-bootstrap-python is so old that it cannot do that >> by itself). >> dnl Packages that depend on actual xz or liblzma should depend on >> the liblzma spkg. >> AS_IF(["$SAGE_BOOTSTRAP_PYTHON" -c "import lzma" 2>& >> AS_MESSAGE_LOG_FD], [ >> sage_require_xz=no >> ]) >> ]) >> >> HTH >> Dima >> >> >>> Of course the other complication is that I need to be able to make the >>> sage installation >>> self-contained and relocatable. I have done a lot of experimentation >>> with that over the >>> last few days, which seems to indicate that it is not possible to make a >>> standard "prefix" >>> installation of python relocatable - one has to use a "framework" >>> installation to do that. >>> >>> - Marc >>> >>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "sage-devel" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sage-devel+...@googlegroups.com. >>> >> To view this discussion visit >>> https://groups.google.com/d/msgid/sage-devel/a88d22ca-b9a4-485a-9912-d140338d8f7cn%40googlegroups.com >>> <https://groups.google.com/d/msgid/sage-devel/a88d22ca-b9a4-485a-9912-d140338d8f7cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/sage-devel/edf5c543-d148-4e29-9f1e-c2219f9e8e1bn%40googlegroups.com > <https://groups.google.com/d/msgid/sage-devel/edf5c543-d148-4e29-9f1e-c2219f9e8e1bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0qSRgYzdoppXKtiZpXJ0XCRsa-Z9yEZVDvcFnM5juJdw%40mail.gmail.com.
