Hi,
On Fri, 11 Aug 2023 at 02:00, Andres Freund <and...@anarazel.de> wrote:
> At the very least this would need to be combined with
>
> commit 950e64fa46b164df87b5eb7c6e15213ab9880f87
> Author: Andres Freund <and...@anarazel.de>
> Date: 2022-07-18 17:06:34 -0700
>
> Use STDOUT/STDERR_FILENO in most of syslogger.
950e64fa46b164df87b5eb7c6e15213ab9880f87 needs to be combined with
92f478657c5544eba560047c39eba8a030ddb83e. So, I combined
59751ae47fd43add30350a4258773537e98d4063,
950e64fa46b164df87b5eb7c6e15213ab9880f87 and
92f478657c5544eba560047c39eba8a030ddb83e in a single commit [1].
v2-0001-windows-Fix-windows-logging-problems.patch is a combination of
[1] and the rest are the same.
Regards,
Nazir Bilal Yavuz
Microsoft
From 213b049f4f62d22e9b87fbaa572f9e35cf518d63 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavu...@gmail.com>
Date: Thu, 10 Aug 2023 18:40:35 +0300
Subject: [PATCH v2 2/3] Make PG_TEST_USE_UNIX_SOCKETS work for tap tests on
windows.
This commit is backpatched version of
45f52709d86ceaaf282a440f6311c51fc526340b for PG 14.
---
src/bin/pg_ctl/t/001_start_stop.pl | 1 +
src/test/perl/PostgresNode.pm | 13 ++++++++++++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index d656a7fe183..110926e40e3 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -35,6 +35,7 @@ print $conf TestLib::slurp_file($ENV{TEMP_CONFIG})
if ($use_unix_sockets)
{
print $conf "listen_addresses = ''\n";
+ $tempdir_short =~ s!\\!/!g if $TestLib::windows_os;
print $conf "unix_socket_directories = '$tempdir_short'\n";
}
else
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index aae02b18d69..ca2902e6430 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -126,7 +126,18 @@ INIT
$use_tcp = !$TestLib::use_unix_sockets;
$test_localhost = "127.0.0.1";
$last_host_assigned = 1;
- $test_pghost = $use_tcp ? $test_localhost : TestLib::tempdir_short;
+ if ($use_tcp)
+ {
+ $test_pghost = $test_localhost;
+ }
+ else
+ {
+ # On windows, replace windows-style \ path separators with / when
+ # putting socket directories either in postgresql.conf or libpq
+ # connection strings, otherwise they are interpreted as escapes.
+ $test_pghost = TestLib::tempdir_short;
+ $test_pghost =~ s!\\!/!g if $TestLib::windows_os;
+ }
$ENV{PGHOST} = $test_pghost;
$ENV{PGDATABASE} = 'postgres';
--
2.40.1
From 01606cfec52fe4af7c61cdda32d1dcf15d4268cb Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Tue, 7 Sep 2021 11:56:13 -0700
Subject: [PATCH v2 1/3] windows: Fix windows logging problems
This commit is a combination of
54213cb30968c6679050c005ebd1363b77d797c5,
950e64fa46b164df87b5eb7c6e15213ab9880f87 and
92f478657c5544eba560047c39eba8a030ddb83e. These three commits need to be
backpatched together.
54213cb30968c6679050c005ebd1363b77d797c5:
Previously pgwin32_is_service() would falsely return true when postgres is
started from somewhere within a service, but not as a service. That is
e.g. always the case with windows docker containers, which some CI services
use to run windows tests in.
When postgres falsely thinks its running as a service, no messages are
writting to stdout / stderr. That can be very confusing and causes a few tests
to fail.
To fix additionally check if stderr is invalid in pgwin32_is_service(). For
that to work in backend processes, pg_ctl is changed to pass down handles so
that postgres can do the same check (otherwise "default" handles are created).
While this problem exists in all branches, there have been no reports by
users, the prospective CI usage currently is only for master, and I am not a
windows expert. So doing the change in only master for now seems the sanest
approach.
Author: Andres Freund <and...@anarazel.de>
Reviewed-By: Magnus Hagander <mag...@hagander.net>
Discussion: https://postgr.es/m/20210305185752.3up5eq2eanb7o...@alap3.anarazel.de
950e64fa46b164df87b5eb7c6e15213ab9880f87:
Use STDOUT/STDERR_FILENO in most of syslogger.
This fixes problems on windows when logging collector is used in a service,
failing with:
FATAL: could not redirect stderr: Bad file descriptor
This is triggered by 76e38b37a5. The problem is that STDOUT/STDERR_FILENO
aren't defined on windows, which lead us to use _fileno(stdout) etc, but that
doesn't work if stdout/stderr are closed.
Author: Andres Freund <and...@anarazel.de>
Reported-By: Sandeep Thakkar <sandeep.thak...@enterprisedb.com>
Message-Id: 20220520164558.ozb7lm6unakqz...@alap3.anarazel.de (on pgsql-packagers)
Backpatch: 15-, where 76e38b37a5 came in
92f478657c5544eba560047c39eba8a030ddb83e:
windows: msvc: Define STDIN/OUT/ERR_FILENO.
Because they are not available we've used _fileno(stdin) in some places, but
that doesn't reliably work, because stdin might be closed. This is the
prerequisite of the subsequent commit, fixing a failure introduced in
76e38b37a5.
Author: Andres Freund <and...@anarazel.de>
Reported-By: Sandeep Thakkar <sandeep.thak...@enterprisedb.com>
Message-Id: 20220520164558.ozb7lm6unakqz...@alap3.anarazel.de (on pgsql-packagers)
Backpatch: 15-, where 76e38b37a5 came in
---
src/backend/postmaster/syslogger.c | 18 +++++++--------
src/bin/pg_ctl/pg_ctl.c | 33 ++++++++++++++++++++++++++++
src/include/port/win32_msvc/unistd.h | 8 +++++++
src/port/win32security.c | 18 ++++++++++++---
4 files changed, 65 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c
index cad43bdef23..0bf1e438f5b 100644
--- a/src/backend/postmaster/syslogger.c
+++ b/src/backend/postmaster/syslogger.c
@@ -196,12 +196,12 @@ SysLoggerMain(int argc, char *argv[])
* if they fail then presumably the file descriptors are closed and
* any writes will go into the bitbucket anyway.
*/
- close(fileno(stdout));
- close(fileno(stderr));
+ close(STDOUT_FILENO);
+ close(STDERR_FILENO);
if (fd != -1)
{
- (void) dup2(fd, fileno(stdout));
- (void) dup2(fd, fileno(stderr));
+ (void) dup2(fd, STDOUT_FILENO);
+ (void) dup2(fd, STDERR_FILENO);
close(fd);
}
}
@@ -213,7 +213,7 @@ SysLoggerMain(int argc, char *argv[])
*/
#ifdef WIN32
else
- _setmode(_fileno(stderr), _O_TEXT);
+ _setmode(STDERR_FILENO, _O_TEXT);
#endif
/*
@@ -675,12 +675,12 @@ SysLogger_Start(void)
#ifndef WIN32
fflush(stdout);
- if (dup2(syslogPipe[1], fileno(stdout)) < 0)
+ if (dup2(syslogPipe[1], STDOUT_FILENO) < 0)
ereport(FATAL,
(errcode_for_file_access(),
errmsg("could not redirect stdout: %m")));
fflush(stderr);
- if (dup2(syslogPipe[1], fileno(stderr)) < 0)
+ if (dup2(syslogPipe[1], STDERR_FILENO) < 0)
ereport(FATAL,
(errcode_for_file_access(),
errmsg("could not redirect stderr: %m")));
@@ -697,12 +697,12 @@ SysLogger_Start(void)
fflush(stderr);
fd = _open_osfhandle((intptr_t) syslogPipe[1],
_O_APPEND | _O_BINARY);
- if (dup2(fd, _fileno(stderr)) < 0)
+ if (dup2(fd, STDERR_FILENO) < 0)
ereport(FATAL,
(errcode_for_file_access(),
errmsg("could not redirect stderr: %m")));
close(fd);
- _setmode(_fileno(stderr), _O_BINARY);
+ _setmode(STDERR_FILENO, _O_BINARY);
/*
* Now we are done with the write end of the pipe.
diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c
index 552e3a6a1c8..ed4ac27fc43 100644
--- a/src/bin/pg_ctl/pg_ctl.c
+++ b/src/bin/pg_ctl/pg_ctl.c
@@ -1773,6 +1773,31 @@ typedef BOOL (WINAPI * __SetInformationJobObject) (HANDLE, JOBOBJECTINFOCLASS, L
typedef BOOL (WINAPI * __AssignProcessToJobObject) (HANDLE, HANDLE);
typedef BOOL (WINAPI * __QueryInformationJobObject) (HANDLE, JOBOBJECTINFOCLASS, LPVOID, DWORD, LPDWORD);
+/*
+ * Set up STARTUPINFO for the new process to inherit this process' handles.
+ *
+ * Process started as services appear to have "empty" handles (GetStdHandle()
+ * returns NULL) rather than invalid ones. But passing down NULL ourselves
+ * doesn't work, it's interpreted as STARTUPINFO->hStd* not being set. But we
+ * can pass down INVALID_HANDLE_VALUE - which makes GetStdHandle() in the new
+ * process (and its child processes!) return INVALID_HANDLE_VALUE. Which
+ * achieves the goal of postmaster running in a similar environment as pg_ctl.
+ */
+static void
+InheritStdHandles(STARTUPINFO* si)
+{
+ si->dwFlags |= STARTF_USESTDHANDLES;
+ si->hStdInput = GetStdHandle(STD_INPUT_HANDLE);
+ if (si->hStdInput == NULL)
+ si->hStdInput = INVALID_HANDLE_VALUE;
+ si->hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
+ if (si->hStdOutput == NULL)
+ si->hStdOutput = INVALID_HANDLE_VALUE;
+ si->hStdError = GetStdHandle(STD_ERROR_HANDLE);
+ if (si->hStdError == NULL)
+ si->hStdError = INVALID_HANDLE_VALUE;
+}
+
/*
* Create a restricted token, a job object sandbox, and execute the specified
* process with it.
@@ -1810,6 +1835,14 @@ CreateRestrictedProcess(char *cmd, PROCESS_INFORMATION *processInfo, bool as_ser
ZeroMemory(&si, sizeof(si));
si.cb = sizeof(si);
+ /*
+ * Set stdin/stdout/stderr handles to be inherited in the child
+ * process. That allows postmaster and the processes it starts to perform
+ * additional checks to see if running in a service (otherwise they get
+ * the default console handles - which point to "somewhere").
+ */
+ InheritStdHandles(&si);
+
Advapi32Handle = LoadLibrary("ADVAPI32.DLL");
if (Advapi32Handle != NULL)
{
diff --git a/src/include/port/win32_msvc/unistd.h b/src/include/port/win32_msvc/unistd.h
index b63f4770a1c..b7795ba03c4 100644
--- a/src/include/port/win32_msvc/unistd.h
+++ b/src/include/port/win32_msvc/unistd.h
@@ -1 +1,9 @@
/* src/include/port/win32_msvc/unistd.h */
+
+/*
+ * MSVC does not define these, nor does _fileno(stdin) etc reliably work
+ * (returns -1 if stdin/out/err are closed).
+ */
+#define STDIN_FILENO 0
+#define STDOUT_FILENO 1
+#define STDERR_FILENO 2
diff --git a/src/port/win32security.c b/src/port/win32security.c
index 4a673fde19a..6a1bd9b8654 100644
--- a/src/port/win32security.c
+++ b/src/port/win32security.c
@@ -95,8 +95,11 @@ pgwin32_is_admin(void)
* We consider ourselves running as a service if one of the following is
* true:
*
- * 1) We are running as LocalSystem (only used by services)
- * 2) Our token contains SECURITY_SERVICE_RID (automatically added to the
+ * 1) Standard error is not valid (always the case for services, and pg_ctl
+ * running as a service "passes" that down to postgres,
+ * c.f. CreateRestrictedProcess())
+ * 2) We are running as LocalSystem (only used by services)
+ * 3) Our token contains SECURITY_SERVICE_RID (automatically added to the
* process token by the SCM when starting a service)
*
* The check for LocalSystem is needed, because surprisingly, if a service
@@ -121,12 +124,21 @@ pgwin32_is_service(void)
PSID ServiceSid;
PSID LocalSystemSid;
SID_IDENTIFIER_AUTHORITY NtAuthority = {SECURITY_NT_AUTHORITY};
+ HANDLE stderr_handle;
/* Only check the first time */
if (_is_service != -1)
return _is_service;
- /* First check for LocalSystem */
+ /* Check if standard error is not valid */
+ stderr_handle = GetStdHandle(STD_ERROR_HANDLE);
+ if (stderr_handle != INVALID_HANDLE_VALUE && stderr_handle != NULL)
+ {
+ _is_service = 0;
+ return _is_service;
+ }
+
+ /* Check if running as LocalSystem */
if (!AllocateAndInitializeSid(&NtAuthority, 1,
SECURITY_LOCAL_SYSTEM_RID, 0, 0, 0, 0, 0, 0, 0,
&LocalSystemSid))
--
2.40.1
From 872531c9f9a2a2df236905d9616fa448939e75b1 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavu...@gmail.com>
Date: Thu, 10 Aug 2023 18:56:06 +0300
Subject: [PATCH v2 3/3] ci: Add PG CI to PG 14
PG CI is added starting from PG 15. Adding PG CI to PG 14 and below
could be beneficial. So, add PG CI to PG 14.
For the initial CI commit see: 93d97349461347d952e8cebdf62f5aa84b4bd20a
---
.cirrus.yml | 616 ++++++++++++++++++++++++
src/tools/ci/README | 67 +++
src/tools/ci/cores_backtrace.sh | 50 ++
src/tools/ci/gcp_freebsd_repartition.sh | 28 ++
src/tools/ci/pg_ci_base.conf | 14 +
src/tools/ci/windows_build_config.pl | 13 +
6 files changed, 788 insertions(+)
create mode 100644 .cirrus.yml
create mode 100644 src/tools/ci/README
create mode 100755 src/tools/ci/cores_backtrace.sh
create mode 100755 src/tools/ci/gcp_freebsd_repartition.sh
create mode 100644 src/tools/ci/pg_ci_base.conf
create mode 100644 src/tools/ci/windows_build_config.pl
diff --git a/.cirrus.yml b/.cirrus.yml
new file mode 100644
index 00000000000..08cb737f81d
--- /dev/null
+++ b/.cirrus.yml
@@ -0,0 +1,616 @@
+# CI configuration file for CI utilizing cirrus-ci.org
+#
+# For instructions on how to enable the CI integration in a repository and
+# further details, see src/tools/ci/README
+
+
+env:
+ # Source of images / containers
+ GCP_PROJECT: pg-ci-images
+ IMAGE_PROJECT: $GCP_PROJECT
+ CONTAINER_REPO: us-docker.pkg.dev/${GCP_PROJECT}/ci
+
+ # The lower depth accelerates git clone. Use a bit of depth so that
+ # concurrent tasks and retrying older jobs have a chance of working.
+ CIRRUS_CLONE_DEPTH: 500
+ # Useful to be able to analyse what in a script takes long
+ CIRRUS_LOG_TIMESTAMP: true
+
+ CCACHE_MAXSIZE: "250M"
+
+ # target to test, for all but windows
+ CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
+ CHECKFLAGS: -Otarget
+ PROVE_FLAGS: --timer
+ PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests
+ TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
+ PG_TEST_EXTRA: kerberos ldap ssl
+
+
+# What files to preserve in case tests fail
+on_failure: &on_failure
+ log_artifacts:
+ paths:
+ - "**/*.log"
+ - "**/*.diffs"
+ - "**/regress_log_*"
+ type: text/plain
+
+task:
+ name: FreeBSD - 13
+
+ env:
+ # FreeBSD on GCP is slow when running with larger number of CPUS /
+ # jobs. Using one more job than cpus seems to work best.
+ CPUS: 2
+ BUILD_JOBS: 3
+ TEST_JOBS: 3
+
+ CCACHE_DIR: /tmp/ccache_dir
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+
+ compute_engine_instance:
+ image_project: $IMAGE_PROJECT
+ image: family/pg-ci-freebsd-13
+ platform: freebsd
+ cpu: $CPUS
+ memory: 4G
+ disk: 50
+
+ sysinfo_script: |
+ id
+ uname -a
+ ulimit -a -H && ulimit -a -S
+ export
+
+ ccache_cache:
+ folder: $CCACHE_DIR
+ # Work around performance issues due to 32KB block size
+ repartition_script: src/tools/ci/gcp_freebsd_repartition.sh
+ create_user_script: |
+ pw useradd postgres
+ chown -R postgres:postgres .
+ mkdir -p ${CCACHE_DIR}
+ chown -R postgres:postgres ${CCACHE_DIR}
+ setup_core_files_script: |
+ mkdir -m 770 /tmp/cores
+ chown root:postgres /tmp/cores
+ sysctl kern.corefile='/tmp/cores/%N.%P.core'
+ setup_additional_packages_script: |
+ #pkg install -y ...
+
+ # NB: Intentionally build without --with-llvm. The freebsd image size is
+ # already large enough to make VM startup slow, and even without llvm
+ # freebsd already takes longer than other platforms except for windows.
+ configure_script: |
+ su postgres <<-EOF
+ ./configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ --with-gssapi \
+ --with-icu \
+ --with-ldap \
+ --with-libxml \
+ --with-libxslt \
+ --with-lz4 \
+ --with-pam \
+ --with-perl \
+ --with-python \
+ --with-ssl=openssl \
+ --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \
+ --with-uuid=bsd \
+ \
+ --with-includes=/usr/local/include \
+ --with-libs=/usr/local/lib \
+ \
+ CC="ccache cc" \
+ CXX="ccache c++" \
+ CFLAGS="-Og -ggdb"
+ EOF
+ build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ upload_caches: ccache
+
+ # The use of script avoids make -Otarget complaints about fcntl() on
+ # platforms without support for locking pipes. See also
+ # https://savannah.gnu.org/bugs/?60774
+ # script uses pseudo-ttys, which do support locking.
+ test_world_script:
+ - su postgres -c "time script test.log gmake -s -j${TEST_JOBS} ${CHECK} ${CHECKFLAGS}"
+
+ on_failure:
+ <<: *on_failure
+ cores_script: src/tools/ci/cores_backtrace.sh freebsd /tmp/cores
+
+
+# configure feature flags, shared between the task running the linux tests and
+# the CompilerWarnings task
+LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >-
+ --with-gssapi
+ --with-icu
+ --with-ldap
+ --with-libxml
+ --with-libxslt
+ --with-llvm
+ --with-lz4
+ --with-pam
+ --with-perl
+ --with-python
+ --with-selinux
+ --with-ssl=openssl
+ --with-systemd
+ --with-tcl --with-tclconfig=/usr/lib/tcl8.6/
+ --with-uuid=ossp
+
+
+task:
+ name: Linux - Debian Bullseye
+
+ env:
+ CPUS: 4
+ BUILD_JOBS: 4
+ TEST_JOBS: 8 # experimentally derived to be a decent choice
+
+ CCACHE_DIR: /tmp/ccache_dir
+ DEBUGINFOD_URLS: "https://debuginfod.debian.net";
+
+ LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+
+ compute_engine_instance:
+ image_project: $IMAGE_PROJECT
+ image: family/pg-ci-bullseye
+ platform: linux
+ cpu: $CPUS
+ memory: 4G
+
+ ccache_cache:
+ folder: ${CCACHE_DIR}
+
+ sysinfo_script: |
+ id
+ uname -a
+ cat /proc/cmdline
+ ulimit -a -H && ulimit -a -S
+ export
+ create_user_script: |
+ useradd -m postgres
+ chown -R postgres:postgres .
+ mkdir -p ${CCACHE_DIR}
+ chown -R postgres:postgres ${CCACHE_DIR}
+ echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf
+ su postgres -c "ulimit -l -H && ulimit -l -S"
+ setup_core_files_script: |
+ mkdir -m 770 /tmp/cores
+ chown root:postgres /tmp/cores
+ sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core'
+ setup_additional_packages_script: |
+ #apt-get update
+ #DEBIAN_FRONTEND=noninteractive apt-get -y install ...
+
+ configure_script: |
+ su postgres <<-EOF
+ ./configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ ${LINUX_CONFIGURE_FEATURES} \
+ \
+ CC="ccache gcc" \
+ CXX="ccache g++" \
+ CLANG="ccache clang" \
+ CFLAGS="-Og -ggdb" \
+ CXXFLAGS="-Og -ggdb"
+ EOF
+ build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ upload_caches: ccache
+
+ test_world_script: |
+ su postgres <<-EOF
+ ulimit -c unlimited # default is 0
+ make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS}
+ EOF
+
+ on_failure:
+ <<: *on_failure
+ cores_script: src/tools/ci/cores_backtrace.sh linux /tmp/cores
+
+
+task:
+ name: macOS - Ventura
+
+ env:
+ CPUS: 4 # always get that much for cirrusci macOS instances
+ BUILD_JOBS: $CPUS
+ # Test performance regresses noticably when using all cores. 8 seems to
+ # work OK. See
+ # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de
+ TEST_JOBS: 8
+
+ CIRRUS_WORKING_DIR: ${HOME}/pgsql/
+ CCACHE_DIR: ${HOME}/ccache
+ HOMEBREW_CACHE: ${HOME}/homebrew-cache
+ PERL5LIB: ${HOME}/perl5/lib/perl5
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+
+ macos_instance:
+ image: ghcr.io/cirruslabs/macos-ventura-base:latest
+
+ sysinfo_script: |
+ id
+ uname -a
+ ulimit -a -H && ulimit -a -S
+ export
+
+ setup_core_files_script:
+ - mkdir ${HOME}/cores
+ - sudo sysctl kern.corefile="${HOME}/cores/core.%P"
+
+ perl_cache:
+ folder: ~/perl5
+ cpan_install_script:
+ - perl -mIPC::Run -e 1 || cpan -T IPC::Run
+ - perl -mIO::Pty -e 1 || cpan -T IO::Pty
+ upload_caches: perl
+
+
+ # XXX: Could we instead install homebrew into a cached directory? The
+ # homebrew installation takes a good bit of time every time, even if the
+ # packages do not need to be downloaded.
+ homebrew_cache:
+ folder: $HOMEBREW_CACHE
+ setup_additional_packages_script: |
+ brew install \
+ ccache \
+ icu4c \
+ krb5 \
+ llvm \
+ lz4 \
+ make \
+ openldap \
+ openssl \
+ python \
+ tcl-tk \
+ zstd
+
+ brew cleanup -s # to reduce cache size
+ upload_caches: homebrew
+
+ ccache_cache:
+ folder: $CCACHE_DIR
+ configure_script: |
+ brewpath="/opt/homebrew"
+ INCLUDES="${brewpath}/include:${INCLUDES}"
+ LIBS="${brewpath}/lib:${LIBS}"
+
+ for pkg in icu4c krb5 openldap openssl zstd ; do
+ pkgpath="${brewpath}/opt/${pkg}"
+ INCLUDES="${pkgpath}/include:${INCLUDES}"
+ LIBS="${pkgpath}/lib:${LIBS}"
+ PKG_CONFIG_PATH="${pkgpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
+ done
+
+ export PKG_CONFIG_PATH
+
+ ./configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ --with-bonjour \
+ --with-gssapi \
+ --with-icu \
+ --with-ldap \
+ --with-libxml \
+ --with-libxslt \
+ --with-lz4 \
+ --with-perl \
+ --with-python \
+ --with-ssl=openssl \
+ --with-tcl --with-tclconfig=${brewpath}/opt/tcl-tk/lib/ \
+ --with-uuid=e2fs \
+ \
+ --prefix=${HOME}/install \
+ --with-includes="${INCLUDES}" \
+ --with-libs="${LIBS}" \
+ \
+ CC="ccache cc" \
+ CXX="ccache c++" \
+ CLANG="ccache ${brewpath}/llvm/bin/ccache" \
+ CFLAGS="-Og -ggdb" \
+ CXXFLAGS="-Og -ggdb" \
+ \
+ LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
+ PYTHON=python3
+ build_script: gmake -s -j${BUILD_JOBS} world-bin
+ upload_caches: ccache
+
+ test_world_script: |
+ ulimit -c unlimited # default is 0
+ ulimit -n 1024 # default is 256, pretty low
+ # See freebsd use of script for explanation
+ script test.log gmake -s -j${TEST_JOBS} ${CHECK} ${CHECKFLAGS}
+
+ on_failure:
+ <<: *on_failure
+ cores_script: src/tools/ci/cores_backtrace.sh macos "${HOME}/cores"
+
+
+task:
+ name: Windows - Server 2019, VS 2019
+
+ env:
+ # Half the allowed per-user CPU cores
+ CPUS: 4
+
+ # Our windows infrastructure doesn't have test concurrency above the level
+ # of a single vcregress test target. Due to that, it's useful to run prove
+ # with multiple jobs. For the other tasks it isn't, because two sources
+ # (make and prove) of concurrency can overload machines.
+ #
+ # The concrete choice of 10 is based on a small bit of experimentation and
+ # likely can be improved upon further.
+ PROVE_FLAGS: -j10 --timer
+
+ # The default cirrus working dir is in a directory msbuild complains about
+ CIRRUS_WORKING_DIR: "c:/cirrus"
+ # Avoid re-installing over and over
+ NO_TEMP_INSTALL: 1
+ # git's tar doesn't deal with drive letters, see
+ # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net
+ TAR: "c:/windows/system32/tar.exe"
+ # Avoids port conflicts between concurrent tap test runs
+ PG_TEST_USE_UNIX_SOCKETS: 1
+ PG_REGRESS_SOCK_DIR: "c:/cirrus/"
+ # -m enables parallelism
+ # verbosity:minimal + Summary reduce verbosity, while keeping a summary of
+ # errors/warnings
+ # ForceNoAlign prevents msbuild from introducing line-breaks for long lines
+ # disable file tracker, we're never going to rebuild, and it slows down the
+ # build
+ MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo
+
+ # If tests hang forever, cirrus eventually times out. In that case log
+ # output etc is not uploaded, making the problem hard to debug. Of course
+ # tests internally should have shorter timeouts, but that's proven to not
+ # be sufficient. 15min currently is fast enough to finish individual test
+ # "suites".
+ T_C: "\"C:/Program Files/Git/usr/bin/timeout.exe\" -v -k60s 15m"
+
+ # startcreate_script starts a postgres instance that we don't want to get
+ # killed at the end of that script (it's stopped in stop_script). Can't
+ # trivially use background_scripts because a) need pg_ctl's dropping of
+ # permissions b) need to wait for startup to have finished, and we don't
+ # currently have a tool for that...
+ CIRRUS_ESCAPING_PROCESSES: 1
+
+ # Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That
+ # prevents crash reporting from working unless binaries do SetErrorMode()
+ # themselves. Furthermore, it appears that either python or, more likely,
+ # the C runtime has a bug where SEM_NOGPFAULTERRORBOX can very
+ # occasionally *trigger* a crash on process exit - which is hard to debug,
+ # given that it explicitly prevents crash dumps from working...
+ # 0x8001 is SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX
+ CIRRUS_WINDOWS_ERROR_MODE: 0x8001
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+
+ compute_engine_instance:
+ image_project: $IMAGE_PROJECT
+ image: family/pg-ci-windows-ci-vs-2019
+ platform: windows
+ cpu: $CPUS
+ memory: 4G
+
+ sysinfo_script: |
+ chcp
+ systeminfo
+ powershell -Command get-psdrive -psprovider filesystem
+ set
+
+ setup_additional_packages_script: |
+ REM choco install -y --no-progress ...
+
+ configure_script:
+ # copy errors out when using forward slashes
+ - copy src\tools\ci\windows_build_config.pl src\tools\msvc\config.pl
+ - vcvarsall x64
+ - perl src/tools/msvc/mkvcbuild.pl
+ build_script:
+ - vcvarsall x64
+ - msbuild %MSBFLAGS% pgsql.sln
+ tempinstall_script:
+ # Installation on windows currently only completely works from src/tools/msvc
+ - cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install
+
+ test_regress_parallel_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl check parallel
+ startcreate_script: |
+ rem paths to binaries need backslashes
+ tmp_install\bin\pg_ctl.exe initdb -D tmp_check/db -l tmp_check/initdb.log --options=--no-sync
+ echo include '%TEMP_CONFIG%' >> tmp_check/db/postgresql.conf
+ tmp_install\bin\pg_ctl.exe start -D tmp_check/db -l tmp_check/postmaster.log
+
+ test_pl_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl plcheck
+ test_isolation_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl isolationcheck
+ test_modules_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl modulescheck
+ test_contrib_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl contribcheck
+ stop_script: |
+ tmp_install\bin\pg_ctl.exe stop -D tmp_check/db -l tmp_check/postmaster.log
+ test_ssl_script: |
+ set with_ssl=openssl
+ %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/
+ test_subscription_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/subscription/
+ test_authentication_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/authentication/
+ test_recovery_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl recoverycheck
+ test_bin_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl bincheck
+ test_ecpg_script: |
+ rem tries to build additional stuff
+ vcvarsall x64
+ rem References ecpg_regression.proj in the current dir
+ cd src/tools/msvc
+ %T_C% perl vcregress.pl ecpgcheck
+
+ on_failure:
+ <<: *on_failure
+ crashlog_artifacts:
+ path: "crashlog-*.txt"
+ type: text/plain
+
+
+task:
+ name: CompilerWarnings
+
+ # To limit unnecessary work only run this once the normal linux test succeeds
+ depends_on:
+ - Linux - Debian Bullseye
+
+ env:
+ CPUS: 4
+ BUILD_JOBS: 4
+
+ # Use larger ccache cache, as this task compiles with multiple compilers /
+ # flag combinations
+ CCACHE_MAXSIZE: "1GB"
+ CCACHE_DIR: "/tmp/ccache_dir"
+
+ LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
+
+ # task that did not run, count as a success, so we need to recheck Linux'
+ # condition here ...
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+
+ container:
+ image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
+ cpu: $CPUS
+
+ sysinfo_script: |
+ id
+ uname -a
+ cat /proc/cmdline
+ ulimit -a -H && ulimit -a -S
+ gcc -v
+ clang -v
+ export
+
+ ccache_cache:
+ folder: $CCACHE_DIR
+
+ setup_additional_packages_script: |
+ #apt-get update
+ #DEBIAN_FRONTEND=noninteractive apt-get -y install ...
+
+ ###
+ # Test that code can be built with gcc/clang without warnings
+ ###
+
+ setup_script: echo "COPT=-Werror" > src/Makefile.custom
+
+ # Trace probes have a history of getting accidentally broken. Use the
+ # different compilers to build with different combinations of dtrace on/off
+ # and cassert on/off.
+
+ # gcc, cassert off, dtrace on
+ always:
+ gcc_warning_script: |
+ time ./configure \
+ --cache gcc.cache \
+ --enable-dtrace \
+ ${LINUX_CONFIGURE_FEATURES} \
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} world-bin
+
+ # gcc, cassert on, dtrace off
+ always:
+ gcc_a_warning_script: |
+ time ./configure \
+ --cache gcc.cache \
+ --enable-cassert \
+ ${LINUX_CONFIGURE_FEATURES} \
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} world-bin
+
+ # clang, cassert off, dtrace off
+ always:
+ clang_warning_script: |
+ time ./configure \
+ --cache clang.cache \
+ ${LINUX_CONFIGURE_FEATURES} \
+ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} world-bin
+
+ # clang, cassert on, dtrace on
+ always:
+ clang_a_warning_script: |
+ time ./configure \
+ --cache clang.cache \
+ --enable-cassert \
+ --enable-dtrace \
+ ${LINUX_CONFIGURE_FEATURES} \
+ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} world-bin
+
+ # cross-compile to windows
+ always:
+ mingw_cross_warning_script: |
+ time ./configure \
+ --host=x86_64-w64-mingw32 \
+ --enable-cassert \
+ CC="ccache x86_64-w64-mingw32-gcc" \
+ CXX="ccache x86_64-w64-mingw32-g++"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} world-bin
+
+ ###
+ # Verify docs can be built
+ ###
+ # XXX: Only do this if there have been changes in doc/ since last build
+ always:
+ docs_build_script: |
+ time ./configure \
+ --cache gcc.cache \
+ CC="ccache gcc" \
+ CXX="ccache g++" \
+ CLANG="ccache clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s -j${BUILD_JOBS} -C doc
+
+ ###
+ # Verify headerscheck / cpluspluscheck succeed
+ #
+ # - Don't use ccache, the files are uncacheable, polluting ccache's
+ # cache
+ # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose
+ # - XXX have to disable ICU to avoid errors:
+ # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de
+ # - XXX: the -Wno-register avoids verbose warnings:
+ # https://postgr.es/m/20220308181837.aun3tdtdvao4vb7o%40alap3.anarazel.de
+ ###
+ always:
+ headers_headerscheck_script: |
+ time ./configure \
+ ${LINUX_CONFIGURE_FEATURES} \
+ --without-icu \
+ --quiet \
+ CC="gcc" CXX"=g++" CLANG="clang"
+ make -s -j${BUILD_JOBS} clean
+ time make -s headerscheck EXTRAFLAGS='-fmax-errors=10'
+ headers_cpluspluscheck_script: |
+ time make -s cpluspluscheck EXTRAFLAGS='-Wno-register -fmax-errors=10'
+
+ always:
+ upload_caches: ccache
diff --git a/src/tools/ci/README b/src/tools/ci/README
new file mode 100644
index 00000000000..6c1cac858f3
--- /dev/null
+++ b/src/tools/ci/README
@@ -0,0 +1,67 @@
+Postgres Continuous Integration (CI)
+====================================
+
+Postgres has two forms of CI:
+
+1) All supported branches in the main postgres repository are continuously
+ tested via the buildfarm. As this covers only the main repository, it
+ cannot be used during development of features.
+
+ For details see https://buildfarm.postgresql.org/
+
+2) For not yet merged development work, CI can be enabled for some git hosting
+ providers. This allows developers to test patches on a number of platforms
+ before they are merged (or even submitted).
+
+
+Configuring CI on personal repositories
+=======================================
+
+Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci
+currently is only available for github.
+
+
+Enabling cirrus-ci in a github repository
+=========================================
+
+To enable cirrus-ci on a repository, go to
+https://github.com/marketplace/cirrus-ci and select "Public
+Repositories". Then "Install it for free" and "Complete order". The next page
+allows to configure which repositories cirrus-ci has access to. Choose the
+relevant repository and "Install".
+
+See also https://cirrus-ci.org/guide/quick-start/
+
+Once enabled on a repository, future commits and pull-requests in that
+repository will automatically trigger CI builds. These are visible from the
+commit history / PRs, and can also be viewed in the cirrus-ci UI at
+https://cirrus-ci.com/github/<username>/<reponame>/
+
+Hint: all build log files are uploaded to cirrus-ci and can be downloaded
+from the "Artifacts" section fron the cirrus-ci UI after clicking into a
+specific task on a build's summary page.
+
+
+Images used for CI
+==================
+
+To keep CI times tolerable, most platforms use pre-generated images. Some
+platforms use containers, others use full VMs. Images for both are generated
+separately from CI runs, otherwise each git repository that is being tested
+would need to build its own set of containers, which would be wasteful (both
+in space and time.
+
+These images are built, on a daily basis, from the specifications in
+github.com/anarazel/pg-vm-images/
+
+
+Controlling CI via commit messages
+==================================
+
+The behavior of CI can be controlled by special content in commit
+messages. Currently the following controls are available:
+
+- ci-os-only: {(freebsd|linux|macos|windows)}
+
+ Only runs CI on operating systems specified. This can be useful when
+ addressing portability issues affecting only a subset of platforms.
diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh
new file mode 100755
index 00000000000..28d3cecfc67
--- /dev/null
+++ b/src/tools/ci/cores_backtrace.sh
@@ -0,0 +1,50 @@
+#! /bin/sh
+
+if [ $# -ne 2 ]; then
+ echo "cores_backtrace.sh <os> <directory>"
+ exit 1
+fi
+
+os=$1
+directory=$2
+
+case $os in
+ freebsd|linux|macos)
+ ;;
+ *)
+ echo "unsupported operating system ${os}"
+ exit 1
+ ;;
+esac
+
+first=1
+for corefile in $(find "$directory" -type f) ; do
+ if [ "$first" -eq 1 ]; then
+ first=0
+ else
+ # to make it easier to separate the different crash reports
+ echo -e '\n\n'
+ fi
+
+ if [ "$os" = 'macos' ]; then
+ lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit'
+ else
+ auxv=$(gdb --quiet --core ${corefile} --batch -ex 'info auxv' 2>/dev/null)
+ if [ $? -ne 0 ]; then
+ echo "could not process ${corefile}"
+ continue
+ fi
+
+ if [ "$os" = 'freebsd' ]; then
+ binary=$(echo "$auxv" | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g")
+ elif [ "$os" = 'linux' ]; then
+ binary=$(echo "$auxv" | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g")
+ else
+ echo 'should not get here'
+ exit 1
+ fi
+
+ echo "dumping ${corefile} for ${binary}"
+ gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" "$binary" "$corefile" 2>/dev/null
+ fi
+done
diff --git a/src/tools/ci/gcp_freebsd_repartition.sh b/src/tools/ci/gcp_freebsd_repartition.sh
new file mode 100755
index 00000000000..2d5e1738998
--- /dev/null
+++ b/src/tools/ci/gcp_freebsd_repartition.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -e
+set -x
+
+# The default filesystem on freebsd gcp images is very slow to run tests on,
+# due to its 32KB block size
+#
+# XXX: It'd probably better to fix this in the image, using something like
+# https://people.freebsd.org/~lidl/blog/re-root.html
+
+# fix backup partition table after resize
+gpart recover da0
+gpart show da0
+# kill swap, so we can delete a partition
+swapoff -a || true
+# (apparently we can only have 4!?)
+gpart delete -i 3 da0
+gpart add -t freebsd-ufs -l data8k -a 4096 da0
+gpart show da0
+newfs -U -b 8192 /dev/da0p3
+
+# Migrate working directory
+du -hs $CIRRUS_WORKING_DIR
+mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig
+mkdir $CIRRUS_WORKING_DIR
+mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR
+cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/
diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf
new file mode 100644
index 00000000000..d8faa9c26c1
--- /dev/null
+++ b/src/tools/ci/pg_ci_base.conf
@@ -0,0 +1,14 @@
+# Tends to produce too many core files, taking a long time
+restart_after_crash = false
+
+# So that tests using the "manually" started postgres on windows can use
+# prepared statements
+max_prepared_transactions = 10
+
+# Settings that make logs more useful
+log_autovacuum_min_duration = 0
+log_checkpoints = true
+log_connections = true
+log_disconnections = true
+log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
+log_lock_waits = true
diff --git a/src/tools/ci/windows_build_config.pl b/src/tools/ci/windows_build_config.pl
new file mode 100644
index 00000000000..59268a0bb60
--- /dev/null
+++ b/src/tools/ci/windows_build_config.pl
@@ -0,0 +1,13 @@
+use strict;
+use warnings;
+
+our $config;
+
+$config->{"tap_tests"} = 1;
+$config->{"asserts"} = 1;
+
+$config->{"openssl"} = "c:/openssl/1.1/";
+$config->{"perl"} = "c:/strawberry/$ENV{DEFAULT_PERL_VERSION}/perl/";
+$config->{"python"} = "c:/python/";
+
+1;
--
2.40.1
