Hi, On Fri, 22 Nov 2024 at 20:28, Andres Freund <and...@anarazel.de> wrote: > The right approach probably is to include the list of packages in the key. A > bit annoying to change, because we'd need to move the list of packages to an > environment variable or file, but doable. I think something like > > env: > MACOS_PACKAGE_LIST: >- > ccache > icu > ... > fingerprint_script: | > ... > echo $MACOS_PACKAGE_LIST > ... > setup_additional_packages_script: | > sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST > > should work?
I think this is a nice solution and it worked successfully [1]. Now, REL_[17 | 16]_* and master branches use the same cache which is different from the REL_15_STABLE branch's cache. In case you want to continue with this, the patches are attached. I merged 'using a loop in the install script' from Thomas' patch and the change above. [1] First run - second run (See cache is used in the second run.) [Master] https://cirrus-ci.com/task/6398434171682816 - https://cirrus-ci.com/task/6460963865493504 [PG 16] https://cirrus-ci.com/task/5697896752873472 - https://cirrus-ci.com/task/4656279002546176 [PG 15] https://cirrus-ci.com/task/5192066743926784 - https://cirrus-ci.com/task/5033544097988608 -- Regards, Nazir Bilal Yavuz Microsoft
From 6420ec51c8953c9746fe2eb77a91202f5ced4ed9 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz <byavu...@gmail.com> Date: Wed, 27 Nov 2024 14:34:43 +0300 Subject: [PATCH v3] ci: PG 17, 16 - Fix cached MacPorts installation management 1. The error reporting of "port setrequested list-of-packages..." changed, hiding errors we were relying on to know if all packages in our list were already installed. Use a loop instead. 2. The cached MacPorts installation was shared between PostgreSQL major branches, though each branch wanted different packages. Add the list of packages to cache key, so that different branches, when tested in one github account/repo such as postgres/postgres, stop fighting with each other, adding and removing packages. Back-patch to 15 where CI began. Author: Thomas Munro <thomas.mu...@gmail.com> Author: Nazir Bilal Yavuz <byavu...@gmail.com> Suggested-by: Andres Freund <and...@anarazel.de> Discussion: https://postgr.es/m/au2uqfuy2nf43nwy2txmc5t2emhwij7kzupygto3d2ffgtrdgr%40ckvrlwyflnh2 ci-os-only: macos --- .cirrus.tasks.yml | 34 +++++++++++++++------------- src/tools/ci/ci_macports_packages.sh | 10 ++++++-- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index d8b7f9d32ab..197270feeaa 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -424,6 +424,20 @@ task: CCACHE_DIR: ${HOME}/ccache MACPORTS_CACHE: ${HOME}/macports-cache + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + CC: ccache cc CXX: ccache c++ CFLAGS: -Og -ggdb @@ -457,26 +471,14 @@ task: macports_cache: folder: ${MACPORTS_CACHE} fingerprint_script: | - # Include the OS major version in the cache key. If the OS image changes - # to a different major version, we need to reinstall. + # Reinstall packages if the OS major version, the list of the packages + # to install or the MacPorts install script changes. sw_vers -productVersion | sed 's/\..*//' - # Also start afresh if we change our MacPorts install script. + echo $MACOS_PACKAGE_LIST md5 src/tools/ci/ci_macports_packages.sh reupload_on_changes: true setup_additional_packages_script: | - sh src/tools/ci/ci_macports_packages.sh \ - ccache \ - icu \ - kerberos5 \ - lz4 \ - meson \ - openldap \ - openssl \ - p5.34-io-tty \ - p5.34-ipc-run \ - python312 \ - tcl \ - zstd + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST # system python doesn't provide headers sudo /opt/local/bin/port select python3 python312 # Make macports install visible for subsequent steps diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 692fa88c325..fb6164f49c3 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -61,9 +61,15 @@ fi # if setting all the required packages as requested fails, we need # to install at least one of them -if ! sudo port setrequested $packages > /dev/null 2>&1 ; then - echo not all required packages installed, doing so now +echo "checking if all required packages are installed" +for package in $packages ; do + if ! sudo port setrequested $package > /dev/null 2>&1 ; then update_cached_image=1 + fi +done +echo "done" +if [ "$update_cached_image" -eq 1 ]; then + echo not all required packages installed, doing so now # to keep the image small, we deleted the ports tree from the image... sudo port selfupdate # XXX likely we'll need some other way to force an upgrade at some -- 2.45.2
From c68d4d177823041b37172ba81647d6bcbb23c26f Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz <byavu...@gmail.com> Date: Wed, 27 Nov 2024 15:47:21 +0300 Subject: [PATCH v3] ci: PG 15 - Fix cached MacPorts installation management 1. The error reporting of "port setrequested list-of-packages..." changed, hiding errors we were relying on to know if all packages in our list were already installed. Use a loop instead. 2. The cached MacPorts installation was shared between PostgreSQL major branches, though each branch wanted different packages. Add the list of packages to cache key, so that different branches, when tested in one github account/repo such as postgres/postgres, stop fighting with each other, adding and removing packages. Back-patch to 15 where CI began. Author: Thomas Munro <thomas.mu...@gmail.com> Author: Nazir Bilal Yavuz <byavu...@gmail.com> Suggested-by: Andres Freund <and...@anarazel.de> Discussion: https://postgr.es/m/au2uqfuy2nf43nwy2txmc5t2emhwij7kzupygto3d2ffgtrdgr%40ckvrlwyflnh2 ci-os-only: macos --- .cirrus.tasks.yml | 34 +++++++++++++++------------- src/tools/ci/ci_macports_packages.sh | 10 ++++++-- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index d75cbc004ee..3d344327452 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -223,6 +223,20 @@ task: CCACHE_DIR: ${HOME}/ccache MACPORTS_CACHE: ${HOME}/macports-cache + MACOS_PACKAGE_LIST: >- + ccache + gmake + icu + kerberos5 + lz4 + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + <<: *macos_task_template only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' @@ -250,26 +264,14 @@ task: macports_cache: folder: ${MACPORTS_CACHE} fingerprint_script: | - # Include the OS major version in the cache key. If the OS image changes - # to a different major version, we need to reinstall. + # Reinstall packages if the OS major version, the list of the packages + # to install or the MacPorts install script changes. sw_vers -productVersion | sed 's/\..*//' - # Also start afresh if we change our MacPorts install script. + echo $MACOS_PACKAGE_LIST md5 src/tools/ci/ci_macports_packages.sh reupload_on_changes: true setup_additional_packages_script: | - sh src/tools/ci/ci_macports_packages.sh \ - ccache \ - gmake \ - icu \ - kerberos5 \ - lz4 \ - openldap \ - openssl \ - p5.34-io-tty \ - p5.34-ipc-run \ - python312 \ - tcl \ - zstd + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST # system python doesn't provide headers sudo /opt/local/bin/port select python3 python312 # Make macports install visible for subsequent steps diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 692fa88c325..fb6164f49c3 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -61,9 +61,15 @@ fi # if setting all the required packages as requested fails, we need # to install at least one of them -if ! sudo port setrequested $packages > /dev/null 2>&1 ; then - echo not all required packages installed, doing so now +echo "checking if all required packages are installed" +for package in $packages ; do + if ! sudo port setrequested $package > /dev/null 2>&1 ; then update_cached_image=1 + fi +done +echo "done" +if [ "$update_cached_image" -eq 1 ]; then + echo not all required packages installed, doing so now # to keep the image small, we deleted the ports tree from the image... sudo port selfupdate # XXX likely we'll need some other way to force an upgrade at some -- 2.45.2
