[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Change by miss-islington : -- pull_requests: +26498 pull_request: https://github.com/python/cpython/pull/28054 ___ Python tracker ___ __

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Change by miss-islington : -- pull_requests: +26497 pull_request: https://github.com/python/cpython/pull/28053 ___ Python tracker ___ __

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Change by miss-islington : -- keywords: +patch nosy: +miss-islington nosy_count: 3.0 -> 4.0 pull_requests: +26496 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/28052 ___ Python tracker

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Ned Deily added the comment: New changeset 71853a73024a98aa38a3c0444fe364dbd9709134 by Tobias Bergkvist in branch 'main': bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (#27251) https://github.com/python/cpython/commit/71853a73024

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Tobias Bergkvist added the comment: I realised that I needed to define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME in the source file myself - as this is not defined after running the configure-script. I've updated the PR and its description to only focus on the legacy/deprecated approach o

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Tobias Bergkvist added the comment: This makes a lot of sense now. Thank you so much for the thorough explanation Ned - and for highlighting where my assumptions were wrong! I didn’t realise I needed to specify MACOSX_DEPLOYMENT_TARGET to enable backwards compatibility. Is there a reason for

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Ned Deily added the comment: > Regarding the "explicit weak linking" when building on MacOS Big Sur and > later; wouldn't this mean that a Big Sur build wouldn't work on Catalina? No, if it is done correctly. I think you are trying to solve the wrong problem here. As Ronald noted earlier,

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Tobias Bergkvist added the comment: Okay, I decided to look into how I could do dynamic loading as you suggested. Here is a POC (executable) for using _dyld_shared_cache_contains_path when available: ``` #include #include void* libsystemb_handle; typedef bool (*_dyld_shared_cache_contains

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Tobias Bergkvist added the comment: You are absolutely right! From the manpage of dlopen(3) on MacOS Big Sur: > dlopen() examines the mach-o file specified by path. If the file is > compatible with the current process and has not already been loaded into the > current process, it is loaded an

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Ronald Oussoren added the comment: The disadvantage of using dlopen is that this function has side effects, and those can affect program behaviour. Because of this I'm against switching to using dlopen to probe for libraries. -- ___ Python tracke

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Tobias Bergkvist added the comment: An alternative to using _dyld_shared_cache_contains_path is to use dlopen to check for library existence (which is what Apple recommends in their change notes: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Ronald Oussoren added the comment: Anyways, the solution for "build on older macOS version, deploy to Big Sur" is to dynamically look for the relevant API (``_dyld_shared_cache_contains_path``) and use it when available. But only in that scenario, the current code path using explicit weak li

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Ronald Oussoren added the comment: The problem with moving from Catalina to Big Sur is a known issue, AFAIK there's an open issue for this. The problem is that Big Sur moved system libraries into a big blob (which Apple calls the shared library cache). Ctypes uses an API that's new in macOS

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

Change by Zachary Ware : -- components: +macOS nosy: +ned.deily, ronaldoussoren ___ Python tracker ___ ___ Python-bugs-list mailing

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

New submission from Tobias Bergkvist : Python-binaries compiled on either Big Sur or Catalina - and moved to the other MacOS-version will not work as expected when code depends on ctypes.util.find_library. Example symptom of this issue: https://github.com/jupyterlab/jupyterlab/issues/9863 I