rebased on b6a5158f9 and 054325c5e
Also, cirrus/freebsd task can run 3x faster with more CPUs.
Subject: [PATCH 21/21] cirrus: run freebsd with more CPUs+RAM
https://cirrus-ci.com/task/4664440120410112
https://cirrus-ci.com/task/5100411110555648
In the past, I gather there was some undiagnosed issue when using more CPUs
(cirrus now enforces >5GB RAM when using 6 CPUs - maybe you tried to use too
little RAM, or maybe hit bad performance involving NUMA?)
https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4h...@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686
>From 0a79d1065d46e47bdd228d064669e28ef284fe25 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Wed, 25 May 2022 21:53:22 -0500
Subject: [PATCH 01/21] cirrus/windows: add compiler_warnings_script
I'm not sure how to write this test in windows shell; it's also not easy to
write it in posix sh, since windows shell is somehow interpretting && and ||...
https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de
See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956
ci-os-only: windows
https://cirrus-ci.com/task/6183879907213312
https://cirrus-ci.com/task/4876271443247104
---
.cirrus.yml | 8 +++++++-
src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)
create mode 100755 src/tools/ci/windows-compiler-warnings
diff --git a/.cirrus.yml b/.cirrus.yml
index f23d6cae552..bcb8d53db78 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -370,7 +370,8 @@ task:
# ForceNoAlign prevents msbuild from introducing line-breaks for long lines
# disable file tracker, we're never going to rebuild, and it slows down the
# build
- MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo
+ # -fileLoggerParameters1: write warnings to msbuild.warn.log.
+ MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log
# If tests hang forever, cirrus eventually times out. In that case log
# output etc is not uploaded, making the problem hard to debug. Of course
@@ -450,6 +451,11 @@ task:
cd src/tools/msvc
%T_C% perl vcregress.pl ecpgcheck
+ # These should be last, so all the important checks are always run
+ always:
+ compiler_warnings_script:
+ - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log
+
on_failure:
<<: *on_failure
crashlog_artifacts:
diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings
new file mode 100755
index 00000000000..d6f9a1fc569
--- /dev/null
+++ b/src/tools/ci/windows-compiler-warnings
@@ -0,0 +1,16 @@
+#! /bin/sh
+# Success if the given file doesn't exist or is empty, else fail
+# This is a separate file only to avoid dealing with windows shell quoting and escaping.
+set -e
+
+fn=$1
+
+if [ -s "$fn" ]
+then
+ # Display the file's content, then exit indicating failure
+ cat "$fn"
+ exit 1
+else
+ # Success
+ exit 0
+fi
--
2.17.1
>From a622c48d6abbae63288a792d4a2107cbad3f7879 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 9 Jan 2022 18:25:02 -0600
Subject: [PATCH 02/21] cirrus/vcregress: test modules/contrib with
NO_INSTALLCHECK=1
--temp-config must be specified with an "=" because otherwise vcregress runs
pg_regress --temp-config test1 test2 [...],
..which means test1 gets eaten as the argument to --temp-config
https://www.postgresql.org/message-id/20220109191649.GL14051%40telsasoft.com
https://www.postgresql.org/message-id/CA%2BhUKGLneD%2Bq%2BE7upHGwn41KGvbxhsKbJ%2BM-y9nvv7_Xjv8Qog%40mail.gmail.com
---
.cirrus.yml | 4 +-
contrib/basic_archive/Makefile | 2 +-
contrib/pg_stat_statements/Makefile | 2 +-
contrib/test_decoding/Makefile | 2 +-
src/test/modules/snapshot_too_old/Makefile | 2 +-
src/test/modules/worker_spi/Makefile | 2 +-
src/tools/msvc/vcregress.pl | 46 +++++++++++++++++++---
7 files changed, 48 insertions(+), 12 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index bcb8d53db78..2dea2d085cc 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -428,9 +428,9 @@ task:
test_isolation_script: |
%T_C% perl src/tools/msvc/vcregress.pl isolationcheck
test_modules_script: |
- %T_C% perl src/tools/msvc/vcregress.pl modulescheck
+ %T_C% perl src/tools/msvc/vcregress.pl modulescheck install
test_contrib_script: |
- %T_C% perl src/tools/msvc/vcregress.pl contribcheck
+ %T_C% perl src/tools/msvc/vcregress.pl contribcheck install
stop_script: |
tmp_install\bin\pg_ctl.exe stop -D tmp_check/db -l tmp_check/postmaster.log
test_ssl_script: |
diff --git a/contrib/basic_archive/Makefile b/contrib/basic_archive/Makefile
index 14d036e1c42..246358973fe 100644
--- a/contrib/basic_archive/Makefile
+++ b/contrib/basic_archive/Makefile
@@ -4,7 +4,7 @@ MODULES = basic_archive
PGFILEDESC = "basic_archive - basic archive module"
REGRESS = basic_archive
-REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/basic_archive/basic_archive.conf
+REGRESS_OPTS = --temp-config=$(top_srcdir)/contrib/basic_archive/basic_archive.conf
NO_INSTALLCHECK = 1
diff --git a/contrib/pg_stat_statements/Makefile b/contrib/pg_stat_statements/Makefile
index edc40c8bbfb..8684ea0be9c 100644
--- a/contrib/pg_stat_statements/Makefile
+++ b/contrib/pg_stat_statements/Makefile
@@ -16,7 +16,7 @@ PGFILEDESC = "pg_stat_statements - execution statistics of SQL statements"
LDFLAGS_SL += $(filter -lm, $(LIBS))
-REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_stat_statements/pg_stat_statements.conf
+REGRESS_OPTS = --temp-config=$(top_srcdir)/contrib/pg_stat_statements/pg_stat_statements.conf
REGRESS = pg_stat_statements oldextversions
# Disabled because these tests require "shared_preload_libraries=pg_stat_statements",
# which typical installcheck users do not have (e.g. buildfarm clients).
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index b2209064790..3d7ec65fb53 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -10,7 +10,7 @@ ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error
-REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
+REGRESS_OPTS = --temp-config=$(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
# Disabled because these tests require "wal_level=logical", which
diff --git a/src/test/modules/snapshot_too_old/Makefile b/src/test/modules/snapshot_too_old/Makefile
index dfb4537f63c..752a0039fdc 100644
--- a/src/test/modules/snapshot_too_old/Makefile
+++ b/src/test/modules/snapshot_too_old/Makefile
@@ -5,7 +5,7 @@
EXTRA_CLEAN = $(pg_regress_clean_files)
ISOLATION = sto_using_cursor sto_using_select sto_using_hash_index
-ISOLATION_OPTS = --temp-config $(top_srcdir)/src/test/modules/snapshot_too_old/sto.conf
+ISOLATION_OPTS = --temp-config=$(top_srcdir)/src/test/modules/snapshot_too_old/sto.conf
# Disabled because these tests require "old_snapshot_threshold" >= 0, which
# typical installcheck users do not have (e.g. buildfarm clients).
diff --git a/src/test/modules/worker_spi/Makefile b/src/test/modules/worker_spi/Makefile
index cbf9b2e37fd..d9f7d9bab6d 100644
--- a/src/test/modules/worker_spi/Makefile
+++ b/src/test/modules/worker_spi/Makefile
@@ -9,7 +9,7 @@ PGFILEDESC = "worker_spi - background worker example"
REGRESS = worker_spi
# enable our module in shared_preload_libraries for dynamic bgworkers
-REGRESS_OPTS = --temp-config $(top_srcdir)/src/test/modules/worker_spi/dynamic.conf
+REGRESS_OPTS = --temp-config=$(top_srcdir)/src/test/modules/worker_spi/dynamic.conf
# Disable installcheck to ensure we cover dynamic bgworkers.
NO_INSTALLCHECK = 1
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index c3729f6be5e..2d6ccd45419 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -426,6 +426,7 @@ sub plcheck
sub subdircheck
{
my $module = shift;
+ my $installcheck = shift || 1;
if ( !-d "$module/sql"
|| !-d "$module/expected"
@@ -435,7 +436,7 @@ sub subdircheck
}
chdir $module;
- my @tests = fetchTests();
+ my @tests = fetchTests($installcheck);
# Leave if no tests are listed in the module.
if (scalar @tests == 0)
@@ -445,12 +446,13 @@ sub subdircheck
}
my @opts = fetchRegressOpts();
+ push @opts, "--temp-instance=tmp_check" if $installcheck == -1;
print "============================================================\n";
print "Checking $module\n";
my @args = (
"$topdir/$Config/pg_regress/pg_regress",
- "--bindir=${topdir}/${Config}/psql",
+ "--bindir=$tmp_installdir/bin",
"--dbname=contrib_regression", @opts, @tests);
print join(' ', @args), "\n";
system(@args);
@@ -460,6 +462,8 @@ sub subdircheck
sub contribcheck
{
+ my $mode = shift || '';
+
chdir "../../../contrib";
my $mstat = 0;
foreach my $module (glob("*"))
@@ -477,12 +481,25 @@ sub contribcheck
my $status = $? >> 8;
$mstat ||= $status;
}
+
+ # As above, but creates new DB instance for each module. For CI.
+ if ($mode eq "install")
+ {
+ foreach my $module (glob("*"))
+ {
+ subdircheck("$module", -1);
+ $mstat ||= $? >> 8;
+ }
+ }
+
exit $mstat if $mstat;
return;
}
sub modulescheck
{
+ my $mode = shift || '';
+
chdir "../../../src/test/modules";
my $mstat = 0;
foreach my $module (glob("*"))
@@ -491,6 +508,17 @@ sub modulescheck
my $status = $? >> 8;
$mstat ||= $status;
}
+
+ # As above, but creates new DB instance for each module. For CI.
+ if ($mode eq "install")
+ {
+ foreach my $module (glob("*"))
+ {
+ subdircheck("$module", -1);
+ $mstat ||= $? >> 8;
+ }
+ }
+
exit $mstat if $mstat;
return;
}
@@ -558,6 +586,7 @@ sub fetchRegressOpts
# option starting with "--".
@opts = grep { !/\$\(/ && /^--/ }
map { (my $x = $_) =~ s/\Q$(top_builddir)\E/\"$topdir\"/; $x; }
+ map { (my $x = $_) =~ s/\Q$(top_srcdir)\E/\"$topdir\"/; $x; }
split(/\s+/, $1);
}
if ($m =~ /^\s*ENCODING\s*=\s*(\S+)/m)
@@ -583,14 +612,19 @@ sub fetchTests
my $m = <$handle>;
close($handle);
my $t = "";
+ my $installcheck = shift || 1;
$m =~ s{\\\r?\n}{}g;
- # A module specifying NO_INSTALLCHECK does not support installcheck,
- # so bypass its run by returning an empty set of tests.
if ($m =~ /^\s*NO_INSTALLCHECK\s*=\s*\S+/m)
{
- return ();
+ # Skip modules marked installcheck unless running installcheck tests.
+ return () if $installcheck == 1;
+ }
+ else
+ {
+ # Skip modules not marked installcheck if running installcheck tests.
+ return () if $installcheck == -1;
}
if ($m =~ /^REGRESS\s*=\s*(.*)$/gm)
@@ -656,6 +690,8 @@ sub usage
"\nOptions for <arg>: (used by check and installcheck)\n",
" serial serial mode\n",
" parallel parallel mode\n",
+ "\nOptions for <arg>: (used by contribcheck and modulescheck)\n",
+ " install also run tests which require a new instance\n",
"\nOption for <arg>: for taptest\n",
" TEST_DIR (required) directory where tests reside\n";
exit(1);
--
2.17.1
>From 9ccf2633e3db568404d25f81cfbc18ae36da8f07 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 19 Feb 2022 13:06:52 -0600
Subject: [PATCH 03/21] set TESTDIR from src/test/perl rather than
Makefile/vcregress
https://www.postgresql.org/message-id/flat/20220219234148.GC9008%40telsasoft.com
These seem most likely to break:
make check -C src/bin/psql
make check -C src/bin/pgbench
make check -C src/test/modules/test_misc
make check -C src/test/modules/libpq_pipeline
PROVE_TESTS=t/027_stream_regress.pl make check -C src/test/recovery
---
src/Makefile.global.in | 9 ++++++---
src/test/perl/PostgreSQL/Test/Utils.pm | 16 +++++++++-------
src/tools/msvc/vcregress.pl | 1 -
3 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 138d66ac006..cff13bb8087 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -451,8 +451,9 @@ echo "+++ tap install-check in $(subdir) +++" && \
rm -rf '$(CURDIR)'/tmp_check && \
$(MKDIR_P) '$(CURDIR)'/tmp_check && \
cd $(srcdir) && \
- TESTDIR='$(CURDIR)' PATH="$(bindir):$(CURDIR):$$PATH" \
+ PATH="$(bindir):$(CURDIR):$$PATH" \
PGPORT='6$(DEF_PGPORT)' top_builddir='$(CURDIR)/$(top_builddir)' \
+ PG_SUBDIR='$(CURDIR)' \
PG_REGRESS='$(CURDIR)/$(top_builddir)/src/test/regress/pg_regress' \
$(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) $(if $(PROVE_TESTS),$(PROVE_TESTS),t/*.pl)
endef
@@ -462,8 +463,9 @@ echo "+++ tap install-check in $(subdir) +++" && \
rm -rf '$(CURDIR)'/tmp_check && \
$(MKDIR_P) '$(CURDIR)'/tmp_check && \
cd $(srcdir) && \
- TESTDIR='$(CURDIR)' PATH="$(bindir):$(CURDIR):$$PATH" \
+ PATH="$(bindir):$(CURDIR):$$PATH" \
PGPORT='6$(DEF_PGPORT)' top_builddir='$(top_builddir)' \
+ PG_SUBDIR='$(CURDIR)' \
PG_REGRESS='$(top_builddir)/src/test/regress/pg_regress' \
$(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) $(if $(PROVE_TESTS),$(PROVE_TESTS),t/*.pl)
endef
@@ -474,7 +476,8 @@ echo "+++ tap check in $(subdir) +++" && \
rm -rf '$(CURDIR)'/tmp_check && \
$(MKDIR_P) '$(CURDIR)'/tmp_check && \
cd $(srcdir) && \
- TESTDIR='$(CURDIR)' $(with_temp_install) PGPORT='6$(DEF_PGPORT)' \
+ $(with_temp_install) PGPORT='6$(DEF_PGPORT)' \
+ PG_SUBDIR='$(CURDIR)' \
PG_REGRESS='$(CURDIR)/$(top_builddir)/src/test/regress/pg_regress' \
$(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) $(if $(PROVE_TESTS),$(PROVE_TESTS),t/*.pl)
endef
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 1ca2cc59170..063a19b1df5 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -189,19 +189,21 @@ INIT
# test may still fail, but it's more likely to report useful facts.
$SIG{PIPE} = 'IGNORE';
- # Determine output directories, and create them. The base path is the
- # TESTDIR environment variable, which is normally set by the invoking
- # Makefile.
- $tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+ my $test_dir = File::Spec->rel2abs($ENV{PG_SUBDIR} || dirname(dirname($0)));
+ my $test_name = basename($0);
+ $test_name =~ s/\.[^.]+$//;
+
+ $ENV{TESTDIR} = $test_dir;
+
+ # Determine output directories, and create them.
+ $tmp_check = "$test_dir/tmp_check";
$log_path = "$tmp_check/log";
mkdir $tmp_check;
mkdir $log_path;
# Open the test log file, whose name depends on the test name.
- $test_logfile = basename($0);
- $test_logfile =~ s/\.[^.]+$//;
- $test_logfile = "$log_path/regress_log_$test_logfile";
+ $test_logfile = "$log_path/regress_log_$test_name";
open my $testlog, '>', $test_logfile
or die "could not open STDOUT to logfile \"$test_logfile\": $!";
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 2d6ccd45419..ae19ff5736b 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -291,7 +291,6 @@ sub tap_check
$ENV{PG_REGRESS} = "$topdir/$Config/pg_regress/pg_regress";
$ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll";
- $ENV{TESTDIR} = "$dir";
my $module = basename $dir;
# add the module build dir as the second element in the PATH
$ENV{PATH} =~ s!;!;$topdir/$Config/$module;!;
--
2.17.1
>From 2878901bafbef4f194bca9133ccc28061990a9c1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 25 Feb 2022 17:23:26 -0600
Subject: [PATCH 04/21] s!also remove PATH.
This essentially reverts commit f4ce6c4d3a30ec3a12c7f64b90a6fc82887ddd7b
NO And partially reverts 6b04abdfc5e0653542ac5d586e639185a8c61a39
XXX: also set PATH=bindir ?
---
src/Makefile.global.in | 6 +++---
src/interfaces/libpq/Makefile | 1 +
src/test/perl/PostgreSQL/Test/Utils.pm | 10 ++++++++++
src/tools/msvc/vcregress.pl | 4 ----
4 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index cff13bb8087..92bf2d20d7c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -439,7 +439,7 @@ ld_library_path_var = LD_LIBRARY_PATH
# need something more here. If not defined then the expansion does
# nothing.
with_temp_install = \
- PATH="$(abs_top_builddir)/tmp_install$(bindir):$(CURDIR):$$PATH" \
+ PATH="$(abs_top_builddir)/tmp_install$(bindir):$$PATH" \
$(call add_to_path,$(strip $(ld_library_path_var)),$(abs_top_builddir)/tmp_install$(libdir)) \
$(with_temp_install_extra)
@@ -451,7 +451,7 @@ echo "+++ tap install-check in $(subdir) +++" && \
rm -rf '$(CURDIR)'/tmp_check && \
$(MKDIR_P) '$(CURDIR)'/tmp_check && \
cd $(srcdir) && \
- PATH="$(bindir):$(CURDIR):$$PATH" \
+ PATH="$(bindir):$$PATH" \
PGPORT='6$(DEF_PGPORT)' top_builddir='$(CURDIR)/$(top_builddir)' \
PG_SUBDIR='$(CURDIR)' \
PG_REGRESS='$(CURDIR)/$(top_builddir)/src/test/regress/pg_regress' \
@@ -463,7 +463,7 @@ echo "+++ tap install-check in $(subdir) +++" && \
rm -rf '$(CURDIR)'/tmp_check && \
$(MKDIR_P) '$(CURDIR)'/tmp_check && \
cd $(srcdir) && \
- PATH="$(bindir):$(CURDIR):$$PATH" \
+ PATH="$(bindir):$$PATH" \
PGPORT='6$(DEF_PGPORT)' top_builddir='$(top_builddir)' \
PG_SUBDIR='$(CURDIR)' \
PG_REGRESS='$(top_builddir)/src/test/regress/pg_regress' \
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index b5fd72a4acf..d536d850c3c 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -143,6 +143,7 @@ install: all installdirs install-lib
test-build:
$(MAKE) -C test all
+# XXX
check: test-build all
PATH="$(CURDIR)/test:$$PATH" && $(prove_check)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 063a19b1df5..c6c4b2fec8a 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -195,6 +195,16 @@ INIT
$ENV{TESTDIR} = $test_dir;
+ if ($PostgreSQL::Test::Utils::windows_os &&
+ $Config{osname} eq 'MSWin32')
+ {
+ $ENV{PATH} =~ s!;!;$test_dir;!;
+ }
+ else
+ {
+ $ENV{PATH} =~ s!:!:$test_dir:!;
+ }
+
# Determine output directories, and create them.
$tmp_check = "$test_dir/tmp_check";
$log_path = "$tmp_check/log";
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index ae19ff5736b..54c31b0e14f 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -291,10 +291,6 @@ sub tap_check
$ENV{PG_REGRESS} = "$topdir/$Config/pg_regress/pg_regress";
$ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll";
- my $module = basename $dir;
- # add the module build dir as the second element in the PATH
- $ENV{PATH} =~ s!;!;$topdir/$Config/$module;!;
-
rmtree('tmp_check');
system(@args);
my $status = $? >> 8;
--
2.17.1
>From 2adf84066379c4b87d48bc186ea87617caaeff1e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Thu, 14 Apr 2022 06:27:07 -0500
Subject: [PATCH 05/21] cirrus/macos: enable various runtime checks
cirrus CI can take a while to be schedule on macos, but the instance always has
many cores, so this is a good platform to enable options which will slow it
down.
See:
https://www.postgresql.org/message-id/20211217193159.pwrelhiyx7kev...@alap3.anarazel.de
https://www.postgresql.org/message-id/20211213211223.vkgg3wwiss2tragj%40alap3.anarazel.de
https://www.postgresql.org/message-id/CAH2-WzmevBhKNEtqX3N-Tkb0gVBHH62C0KfeTxXzqYES_PiFiA%40mail.gmail.com
https://www.postgresql.org/message-id/20220325000933.vgazz7pjk2ytj...@alap3.anarazel.de
ci-os-only: macos
---
.cirrus.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index 2dea2d085cc..92f41a55271 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -321,6 +321,7 @@ task:
CLANG="ccache ${brewpath}/llvm/bin/ccache" \
CFLAGS="-Og -ggdb" \
CXXFLAGS="-Og -ggdb" \
+ CPPFLAGS="-DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST" \
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
--
2.17.1
>From 1a6fbabbcdc6a0cb196f5b496d663907d6c9a347 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 2 Apr 2022 18:01:39 -0500
Subject: [PATCH 06/21] cirrus/linux: compile with -fsanitize
One concern about this is that the CompilerWarnings task depends on it, and it
seems important for cfbot to not slow down the Linux task so much that the
Linux+Warnings takes longer than windows.
ci-os-only: linux
---
.cirrus.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 92f41a55271..84da09be91e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -203,8 +203,8 @@ task:
CC="ccache gcc" \
CXX="ccache g++" \
CLANG="ccache clang" \
- CFLAGS="-Og -ggdb" \
- CXXFLAGS="-Og -ggdb"
+ CFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all" \
+ CXXFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all"
EOF
build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
upload_caches: ccache
--
2.17.1
>From 08db0a07a11cfbf18e99ae994e3b377fd74de61a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 13 Feb 2022 17:56:40 -0600
Subject: [PATCH 07/21] cirrus/windows: increase timeout to 25min
---
.cirrus.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 84da09be91e..f2bb05266cc 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -377,9 +377,9 @@ task:
# If tests hang forever, cirrus eventually times out. In that case log
# output etc is not uploaded, making the problem hard to debug. Of course
# tests internally should have shorter timeouts, but that's proven to not
- # be sufficient. 15min currently is fast enough to finish individual test
+ # be sufficient. 25min currently is fast enough to finish individual test
# "suites".
- T_C: "\"C:/Program Files/Git/usr/bin/timeout.exe\" -v -k60s 15m"
+ T_C: "\"C:/Program Files/Git/usr/bin/timeout.exe\" -v -k60s 25m"
# startcreate_script starts a postgres instance that we don't want to get
# killed at the end of that script (it's stopped in stop_script). Can't
--
2.17.1
>From 618f91e115d576c2b48b667fe715ef3c1bf5b28a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 25 Feb 2022 17:00:33 -0600
Subject: [PATCH 08/21] vcregress: add alltaptests
https://www.postgresql.org/message-id/20220326031353.gg28...@telsasoft.com
https://www.postgresql.org/message-id/flat/87a81b91-87bf-c0bc-7e4f-06dffadcf...@dunslane.net
In passing, document taptest PROVE_FLAGS
ci-os-only: windows
See also:
d835dd6685246f0737ca42ab68242210681bb220
13d856e177e69083f543d6383eeda9e12ce3c55c
fed6df486dca1b9e53d3f560031b9a236c99f4bb
---
.cirrus.yml | 10 ++-------
src/tools/msvc/vcregress.pl | 43 +++++++++++++++++++++++++++++++++----
2 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f2bb05266cc..ea1f493923a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -437,14 +437,8 @@ task:
test_ssl_script: |
set with_ssl=openssl
%T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/
- test_subscription_script: |
- %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/subscription/
- test_authentication_script: |
- %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/authentication/
- test_recovery_script: |
- %T_C% perl src/tools/msvc/vcregress.pl recoverycheck
- test_bin_script: |
- %T_C% perl src/tools/msvc/vcregress.pl bincheck
+ test_tap_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl alltaptests
test_ecpg_script: |
rem tries to build additional stuff
vcvarsall x64
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 54c31b0e14f..5dcc3419ad7 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -51,7 +51,7 @@ if (-e "src/tools/msvc/buildenv.pl")
my $what = shift || "";
if ($what =~
- /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest)$/i
+ /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest|alltaptests)$/i
)
{
$what = uc $what;
@@ -109,6 +109,7 @@ my %command = (
BINCHECK => \&bincheck,
RECOVERYCHECK => \&recoverycheck,
UPGRADECHECK => \&upgradecheck, # no-op
+ ALLTAPTESTS => \&alltaptests,
TAPTEST => \&taptest,);
my $proc = $command{$what};
@@ -291,6 +292,9 @@ sub tap_check
$ENV{PG_REGRESS} = "$topdir/$Config/pg_regress/pg_regress";
$ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll";
+ print "============================================================\n";
+ print "Checking $dir: @args\n";
+
rmtree('tmp_check');
system(@args);
my $status = $? >> 8;
@@ -305,8 +309,7 @@ sub bincheck
my $mstat = 0;
- # Find out all the existing TAP tests by looking for t/ directories
- # in the tree.
+ # Find the TAP tests by looking for t/ directories
my @bin_dirs = glob("$topdir/src/bin/*");
# Process each test
@@ -321,6 +324,36 @@ sub bincheck
return;
}
+sub alltaptests
+{
+ InstallTemp();
+
+ my $mstat = 0;
+
+ # Find out all the existing TAP tests by looking for t/ directories
+ # in the tree.
+ my @tap_dirs = ();
+ my @top_dir = ($topdir);
+ File::Find::find(
+ { wanted => sub {
+ /^t\z/s
+ && $File::Find::name !~ /\/(kerberos|ldap|ssl|ssl_passphrase_callback)\// # opt-in: warn about these?
+ && push(@tap_dirs, $File::Find::name);
+ }
+ },
+ @top_dir);
+
+ # Process each test
+ foreach my $test_path (@tap_dirs)
+ {
+ my $dir = dirname($test_path);
+ my $status = tap_check($dir);
+ $mstat ||= $status;
+ }
+ exit $mstat if $mstat;
+ return;
+}
+
sub taptest
{
my $dir = shift;
@@ -671,6 +704,7 @@ sub usage
print STDERR
"Usage: vcregress.pl <mode> [<arg>]\n\n",
"Options for <mode>:\n",
+ " alltaptests run all tap tests (except kerberos, ldap, ssl, ssl_passphrase_callback)\n",
" bincheck run tests of utilities in src/bin/\n",
" check deploy instance and run regression tests on it\n",
" contribcheck run tests of modules in contrib/\n",
@@ -688,6 +722,7 @@ sub usage
"\nOptions for <arg>: (used by contribcheck and modulescheck)\n",
" install also run tests which require a new instance\n",
"\nOption for <arg>: for taptest\n",
- " TEST_DIR (required) directory where tests reside\n";
+ " TEST_DIR (required) directory where tests reside\n",
+ " PROVE_FLAGS flags to pass to prove\n";
exit(1);
}
--
2.17.1
>From aec991d1e46783880f7775476330f0e8535cf634 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Thu, 31 Mar 2022 23:52:10 -0500
Subject: [PATCH 09/21] tmp: run tap tests first
ci-os-only: windows
---
.cirrus.yml | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ea1f493923a..ff8b138b9d1 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -416,6 +416,11 @@ task:
# Installation on windows currently only completely works from src/tools/msvc
- cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install
+ this_tap_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl taptest src/interfaces/libpq
+ test_tap_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl alltaptests
+
test_regress_parallel_script: |
%T_C% perl src/tools/msvc/vcregress.pl check parallel
startcreate_script: |
@@ -437,8 +442,6 @@ task:
test_ssl_script: |
set with_ssl=openssl
%T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/
- test_tap_script: |
- %T_C% perl src/tools/msvc/vcregress.pl alltaptests
test_ecpg_script: |
rem tries to build additional stuff
vcvarsall x64
--
2.17.1
>From b84ebc8ce9f600c2be1ba2d6203cc9ca44b0e376 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 19 Feb 2022 13:41:36 -0600
Subject: [PATCH 10/21] vcregress: run alltaptests in parallel
ci-os-only: windows
The test changes are needed to avoid these failures:
https://github.com/justinpryzby/postgres/runs/5174636590
[15:59:59.408] Bailout called. Further testing stopped: could not create test directory "c:/cirrus/tmp_check/t_C:\cirrus\src\bin\pgbench\t\002_pgbench_no_server_stuff": Invalid argument
[15:59:59.408] FAILED--Further testing stopped: could not create test directory "c:/cirrus/tmp_check/t_C:\cirrus\src\bin\pgbench\t\002_pgbench_no_server_stuff": Invalid argument
https://github.com/justinpryzby/postgres/runs/5174788205
[16:37:09.891] # Failed test 'reading traces/disallowed_in_pipeline.trace: could not open "traces/disallowed_in_pipeline.trace": The system cannot find the path specified at C:\cirrus\src\test\modules\libpq_pipeline\t\001_libpq_pipeline.pl line 70.
https://github.com/justinpryzby/postgres/runs/5174877506
pg_regress: could not open file "../regress/parallel_schedule" for reading: No such file or directory
XXX: avoid breaking src/bin/psql/t/010_tab_completion.pl:
See also:
f4ce6c4d3a30ec3a12c7f64b90a6fc82887ddd7b
795862c280c5949bafcd8c44543d887fd32b590a
db973ffb3ca43e65a0bf15175a35184a53bf977d
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 2 +-
src/bin/pgbench/t/002_pgbench_no_server.pl | 4 ++--
.../modules/libpq_pipeline/t/001_libpq_pipeline.pl | 3 ++-
src/test/modules/test_misc/t/003_check_guc.pl | 2 +-
src/test/recovery/t/027_stream_regress.pl | 5 +++--
src/tools/msvc/vcregress.pl | 14 ++++----------
6 files changed, 13 insertions(+), 17 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 2f9b13bf0ae..757f4c76f7d 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -64,7 +64,7 @@ $oldnode->init(extra => [ '--wal-segsize', '1', '--allow-group-access' ]);
$oldnode->start;
# The default location of the source code is the root of this directory.
-my $srcdir = abs_path("../../..");
+my $srcdir = abs_path("$ENV{TESTDIR}/../../..");
# Set up the data of the old instance with a dump or pg_regress.
if (defined($ENV{olddump}))
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index 50bde7dd0fc..bbbb49bb23d 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -8,12 +8,12 @@
use strict;
use warnings;
+use File::Basename;
use PostgreSQL::Test::Utils;
use Test::More;
# create a directory for scripts
-my $testname = $0;
-$testname =~ s,.*/,,;
+my $testname = basename($0);
$testname =~ s/\.pl$//;
my $testdir = "$PostgreSQL::Test::Utils::tmp_check/t_${testname}_stuff";
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
index 0821329c8d3..7e6752562fd 100644
--- a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -50,7 +50,8 @@ for my $testname (@tests)
my $expected;
my $result;
- $expected = slurp_file_eval("traces/$testname.trace");
+ my $inputdir = "$ENV{'TESTDIR'}";
+ $expected = slurp_file_eval("$inputdir/traces/$testname.trace");
next unless $expected ne "";
$result = slurp_file_eval($traceout);
next unless $result ne "";
diff --git a/src/test/modules/test_misc/t/003_check_guc.pl b/src/test/modules/test_misc/t/003_check_guc.pl
index 60459ef759e..167b6ea44b7 100644
--- a/src/test/modules/test_misc/t/003_check_guc.pl
+++ b/src/test/modules/test_misc/t/003_check_guc.pl
@@ -35,7 +35,7 @@ my @not_in_sample_array = split("\n", lc($not_in_sample));
# TAP tests are executed in the directory of the test, in the source tree,
# even for VPATH builds, so rely on that to find postgresql.conf.sample.
-my $rootdir = "../../../..";
+my $rootdir = "$ENV{TESTDIR}/../../../..";
my $sample_file = "$rootdir/src/backend/utils/misc/postgresql.conf.sample";
# List of all the GUCs found in the sample file.
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 7982ac08d0a..c062877bec6 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -70,10 +70,11 @@ my $rc =
. $node_primary->host . " "
. "--port="
. $node_primary->port . " "
- . "--schedule=../regress/parallel_schedule "
+ . "--schedule=\"$dlpath/parallel_schedule\" "
. "--max-concurrent-tests=20 "
- . "--inputdir=../regress "
+ . "--inputdir=\"$dlpath\" "
. "--outputdir=\"$outputdir\"");
+
if ($rc != 0)
{
# Dump out the regression diffs file, if there is one
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 5dcc3419ad7..49168e298ac 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -328,8 +328,6 @@ sub alltaptests
{
InstallTemp();
- my $mstat = 0;
-
# Find out all the existing TAP tests by looking for t/ directories
# in the tree.
my @tap_dirs = ();
@@ -343,14 +341,10 @@ sub alltaptests
},
@top_dir);
- # Process each test
- foreach my $test_path (@tap_dirs)
- {
- my $dir = dirname($test_path);
- my $status = tap_check($dir);
- $mstat ||= $status;
- }
- exit $mstat if $mstat;
+ # Run all the tap tests in a single prove instance for better performance
+ $ENV{PROVE_TESTS} = "@tap_dirs";
+ my $status = tap_check('PROVE_FLAGS=--ext=.pl', "$topdir");
+ exit $status if $status;
return;
}
--
2.17.1
>From 0bd5969a900d7859d5624610db7f118dceb287b8 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 20 Feb 2022 14:05:27 -0600
Subject: [PATCH 11/21] cirrus: make DebugInformationFormat=OldStyle for CI
builds..
This is a hack to allow it to use an alternate setting, since the existing
setting precludes use of ccache/clcache.
Backtraces still seem to work fine.
ci-os-only: windows
---
.cirrus.yml | 2 ++
src/tools/msvc/MSBuildProject.pm | 9 ++++++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ff8b138b9d1..02f04aa98bb 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -374,6 +374,8 @@ task:
# -fileLoggerParameters1: write warnings to msbuild.warn.log.
MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log
+ DebugInformationFormat: OldStyle
+
# If tests hang forever, cirrus eventually times out. In that case log
# output etc is not uploaded, making the problem hard to debug. Of course
# tests internally should have shorter timeouts, but that's proven to not
diff --git a/src/tools/msvc/MSBuildProject.pm b/src/tools/msvc/MSBuildProject.pm
index f24d9e53482..264b98d1f68 100644
--- a/src/tools/msvc/MSBuildProject.pm
+++ b/src/tools/msvc/MSBuildProject.pm
@@ -81,13 +81,15 @@ EOF
</PropertyGroup>
EOF
+ my $debugformat = $ENV{'DebugInformationFormat'} || 'ProgramDatabase';
$self->WriteItemDefinitionGroup(
$f, 'Debug',
{
defs => "_DEBUG;DEBUG=1",
opt => 'Disabled',
strpool => 'false',
- runtime => 'MultiThreadedDebugDLL'
+ runtime => 'MultiThreadedDebugDLL',
+ debugformat => $debugformat,
});
$self->WriteItemDefinitionGroup(
$f,
@@ -96,7 +98,8 @@ EOF
defs => "",
opt => 'Full',
strpool => 'true',
- runtime => 'MultiThreadedDLL'
+ runtime => 'MultiThreadedDLL',
+ debugformat => $debugformat,
});
return;
}
@@ -333,7 +336,7 @@ sub WriteItemDefinitionGroup
<BrowseInformation>false</BrowseInformation>
<WarningLevel>Level3</WarningLevel>
<SuppressStartupBanner>true</SuppressStartupBanner>
- <DebugInformationFormat>ProgramDatabase</DebugInformationFormat>
+ <DebugInformationFormat>$p->{debugformat}</DebugInformationFormat>
<CompileAs>Default</CompileAs>
</ClCompile>
<Link>
--
2.17.1
>From c6ec99c5d674872df8d89b6b9a90b16b8eb63c34 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Wed, 25 May 2022 22:05:13 -0500
Subject: [PATCH 12/21] WIP: cirrus/windows: ccache
https://www.postgresql.org/message-id/flat/20220522232606.GZ19626%40telsasoft.com
Note that cache misses are *several times* slower than running the compiler
directly, so maybe this should be "opt-in" but not used by cfbot with msbuild.
https://cirrus-ci.com/task/5860173364330496 build: 12min 44sec
https://cirrus-ci.com/task/5802653564272640 build: 1min 39sec
ci-os-only: windows, windows-msvc
---
.cirrus.yml | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 02f04aa98bb..50a34373647 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -372,7 +372,10 @@ task:
# disable file tracker, we're never going to rebuild, and it slows down the
# build
# -fileLoggerParameters1: write warnings to msbuild.warn.log.
- MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log
+ MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log /p:UseMultiToolTask=true /p:EnforceProcessCountAcrossBuilds=true /p:CLToolPath=c:\ProgramData\chocolatey\bin
+
+ CCACHE_DIR: $CIRRUS_WORKING_DIR/.ccache
+ CCACHE_LOGFILE: ccache.log
DebugInformationFormat: OldStyle
@@ -396,6 +399,7 @@ task:
image: $CONTAINER_REPO/windows_ci_vs_2019:latest
cpu: $CPUS
memory: 4G
+ greedy: true
sysinfo_script: |
chcp
@@ -403,8 +407,12 @@ task:
powershell -Command get-psdrive -psprovider filesystem
set
- setup_additional_packages_script: |
- REM choco install -y --no-progress ...
+ setup_additional_packages_script:
+ - choco install -y --no-progress ccache --version 4.6.1
+ - cp c:\ProgramData\chocolatey\lib\ccache\tools\ccache-4.6.1-windows-x86_64\ccache.exe \ProgramData\chocolatey\bin\cl.exe
+
+ ccache_cache:
+ folder: $CCACHE_DIR
configure_script:
# copy errors out when using forward slashes
@@ -412,8 +420,13 @@ task:
- vcvarsall x64
- perl src/tools/msvc/mkvcbuild.pl
build_script:
+ #- ccache --clear
- vcvarsall x64
- msbuild %MSBFLAGS% pgsql.sln
+
+ always:
+ upload_caches: ccache
+
tempinstall_script:
# Installation on windows currently only completely works from src/tools/msvc
- cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install
--
2.17.1
>From d3b858d36362cf5747ab7de0086289d233092e6c Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 3 Apr 2022 00:10:20 -0500
Subject: [PATCH 13/21] cirrus/ccache: disable compression and show stats
ci-os-only: linux, windows
ccache since 4.0 enables zstd compression by default.
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
---
.cirrus.yml | 34 ++++++++++++++++++++++++++++++----
1 file changed, 30 insertions(+), 4 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 50a34373647..1d16594803f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,11 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +212,11 @@ task:
CFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all" \
CXXFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -325,7 +335,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -422,7 +436,9 @@ task:
build_script:
#- ccache --clear
- vcvarsall x64
+ - ccache --zero-stats
- msbuild %MSBFLAGS% pgsql.sln
+ - ccache --show-stats
always:
upload_caches: ccache
@@ -537,7 +553,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -548,7 +566,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -558,7 +578,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -570,7 +592,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -581,7 +605,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
>From bf7ac2c24d417886ae97c6e3989c1508d554036f Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Thu, 26 May 2022 22:24:41 -0500
Subject: [PATCH 14/21] another way to install uri-regress/uri_regress
This reverts commit a17fd67d2f2861ae0ce00d1aeefdf2facc47cd5e.
TODO: partially revert 6b04abdfc5e0653542ac5d586e639185a8c61a39
See also:
a17fd67d2f2861ae0ce00d1aeefdf2facc47cd5e
https://www.postgresql.org/message-id/20220416144454.GX26620%40telsasoft.com
https://www.postgresql.org/message-id/20220223211606.ikcwoolsz2nohsw6%40alap3.anarazel.de
TODO: doc/src/sgml/extend.sgml
ci-os-only: linux, windows
---
src/interfaces/libpq/test/Makefile | 8 +++----
src/tools/msvc/Mkvcbuild.pm | 36 +++++++++++++-----------------
2 files changed, 19 insertions(+), 25 deletions(-)
diff --git a/src/interfaces/libpq/test/Makefile b/src/interfaces/libpq/test/Makefile
index 75ac08f943d..91c8d87d528 100644
--- a/src/interfaces/libpq/test/Makefile
+++ b/src/interfaces/libpq/test/Makefile
@@ -14,11 +14,11 @@ endif
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = libpq_testclient libpq_uri_regress
+PROGRAMS = libpq_testclient libpq_uri_regress
-all: $(PROGS)
+all: $(PROGRAMS)
-$(PROGS): $(WIN32RES)
+$(PROGRAMS): $(WIN32RES)
clean distclean maintainer-clean:
- rm -f $(PROGS) *.o
+ rm -f $(PROGRAMS) *.o
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e4feda10fd8..1de77075724 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -37,11 +37,14 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = {};
my @contrib_uselibpq = ();
-my @contrib_uselibpgport = ();
+my @contrib_uselibpgport = ('libpq_uri_regress', 'libpq_testclient');
my @contrib_uselibpgcommon = ();
my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = {};
-my $contrib_extrasource = {};
+my $contrib_extrasource = {
+ 'libpq_uri_regress' => ['src/interfaces/libpq/test/libpq_uri_regress.c'],
+ 'libpq_testclient' => ['src/interfaces/libpq/test/libpq_testclient.c'],
+};
my @contrib_excludes = (
'bool_plperl', 'commit_ts',
'hstore_plperl', 'hstore_plpython',
@@ -284,24 +287,6 @@ sub mkvcbuild
$libpqwalreceiver->AddIncludeDir('src/interfaces/libpq');
$libpqwalreceiver->AddReference($postgres, $libpq);
- my $libpq_testclient =
- $solution->AddProject('libpq_testclient', 'exe', 'misc',
- 'src/interfaces/libpq/test');
- $libpq_testclient->AddFile(
- 'src/interfaces/libpq/test/libpq_testclient.c');
- $libpq_testclient->AddIncludeDir('src/interfaces/libpq');
- $libpq_testclient->AddReference($libpgport, $libpq);
- $libpq_testclient->AddLibrary('ws2_32.lib');
-
- my $libpq_uri_regress =
- $solution->AddProject('libpq_uri_regress', 'exe', 'misc',
- 'src/interfaces/libpq/test');
- $libpq_uri_regress->AddFile(
- 'src/interfaces/libpq/test/libpq_uri_regress.c');
- $libpq_uri_regress->AddIncludeDir('src/interfaces/libpq');
- $libpq_uri_regress->AddReference($libpgport, $libpq);
- $libpq_uri_regress->AddLibrary('ws2_32.lib');
-
my $pgoutput = $solution->AddProject('pgoutput', 'dll', '',
'src/backend/replication/pgoutput');
$pgoutput->AddReference($postgres);
@@ -475,7 +460,7 @@ sub mkvcbuild
push @contrib_excludes, 'uuid-ossp';
}
- foreach my $subdir ('contrib', 'src/test/modules')
+ foreach my $subdir ('contrib', 'src/test/modules', 'src/interfaces/libpq')
{
opendir($D, $subdir) || croak "Could not opendir on $subdir!\n";
while (my $d = readdir($D))
@@ -986,6 +971,15 @@ sub AddContrib
AdjustContribProj($proj);
push @projects, $proj;
}
+ elsif ($mf =~ /^PROGRAMS\s*=\s*(.*)$/mg)
+ {
+ foreach my $proj (split /\s+/, $1)
+ {
+ my $p = $solution->AddProject($proj, 'exe', 'contrib', "$subdir/$n");
+ AdjustContribProj($p);
+ push @projects, $p;
+ }
+ }
else
{
croak "Could not determine contrib module type for $n\n";
--
2.17.1
>From 958998d0798d8851ea6a9fcbae26f350e9cd358e Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Thu, 24 Feb 2022 08:27:41 -0800
Subject: [PATCH 15/21] Move libpq_pipeline test into src/interfaces/libpq.
https://www.postgresql.org/message-id/20220227014626.vi7g7km554gmoape%40alap3.anarazel.de
ci-os-only: windows, linux
---
.../libpq/t/002_pipeline.pl} | 2 +-
src/interfaces/libpq/test/.gitignore | 1 +
src/interfaces/libpq/test/Makefile | 2 +-
.../libpq/test}/libpq_pipeline.c | 0
.../test}/traces/disallowed_in_pipeline.trace | 0
.../libpq/test}/traces/multi_pipelines.trace | 0
.../libpq/test}/traces/nosync.trace | 0
.../libpq/test}/traces/pipeline_abort.trace | 0
.../libpq/test}/traces/pipeline_idle.trace | 0
.../libpq/test}/traces/prepared.trace | 0
.../libpq/test}/traces/simple_pipeline.trace | 0
.../libpq/test}/traces/singlerow.trace | 0
.../libpq/test}/traces/transaction.trace | 0
src/test/modules/Makefile | 1 -
src/test/modules/libpq_pipeline/.gitignore | 5 ----
src/test/modules/libpq_pipeline/Makefile | 25 -------------------
src/test/modules/libpq_pipeline/README | 1 -
src/tools/msvc/Mkvcbuild.pm | 5 ++--
18 files changed, 6 insertions(+), 36 deletions(-)
rename src/{test/modules/libpq_pipeline/t/001_libpq_pipeline.pl => interfaces/libpq/t/002_pipeline.pl} (97%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/libpq_pipeline.c (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/disallowed_in_pipeline.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/multi_pipelines.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/nosync.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/pipeline_abort.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/pipeline_idle.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/prepared.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/simple_pipeline.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/singlerow.trace (100%)
rename src/{test/modules/libpq_pipeline => interfaces/libpq/test}/traces/transaction.trace (100%)
delete mode 100644 src/test/modules/libpq_pipeline/.gitignore
delete mode 100644 src/test/modules/libpq_pipeline/Makefile
delete mode 100644 src/test/modules/libpq_pipeline/README
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/interfaces/libpq/t/002_pipeline.pl
similarity index 97%
rename from src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
rename to src/interfaces/libpq/t/002_pipeline.pl
index 7e6752562fd..0e905061e9f 100644
--- a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
+++ b/src/interfaces/libpq/t/002_pipeline.pl
@@ -50,7 +50,7 @@ for my $testname (@tests)
my $expected;
my $result;
- my $inputdir = "$ENV{'TESTDIR'}";
+ my $inputdir = "$ENV{'TESTDIR'}/test";
$expected = slurp_file_eval("$inputdir/traces/$testname.trace");
next unless $expected ne "";
$result = slurp_file_eval($traceout);
diff --git a/src/interfaces/libpq/test/.gitignore b/src/interfaces/libpq/test/.gitignore
index 6ba78adb678..5c1a281d679 100644
--- a/src/interfaces/libpq/test/.gitignore
+++ b/src/interfaces/libpq/test/.gitignore
@@ -1,2 +1,3 @@
/libpq_testclient
/libpq_uri_regress
+/libpq_pipeline
diff --git a/src/interfaces/libpq/test/Makefile b/src/interfaces/libpq/test/Makefile
index 91c8d87d528..bd459062434 100644
--- a/src/interfaces/libpq/test/Makefile
+++ b/src/interfaces/libpq/test/Makefile
@@ -14,7 +14,7 @@ endif
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGRAMS = libpq_testclient libpq_uri_regress
+PROGRAMS = libpq_testclient libpq_uri_regress libpq_pipeline
all: $(PROGRAMS)
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/interfaces/libpq/test/libpq_pipeline.c
similarity index 100%
rename from src/test/modules/libpq_pipeline/libpq_pipeline.c
rename to src/interfaces/libpq/test/libpq_pipeline.c
diff --git a/src/test/modules/libpq_pipeline/traces/disallowed_in_pipeline.trace b/src/interfaces/libpq/test/traces/disallowed_in_pipeline.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/disallowed_in_pipeline.trace
rename to src/interfaces/libpq/test/traces/disallowed_in_pipeline.trace
diff --git a/src/test/modules/libpq_pipeline/traces/multi_pipelines.trace b/src/interfaces/libpq/test/traces/multi_pipelines.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/multi_pipelines.trace
rename to src/interfaces/libpq/test/traces/multi_pipelines.trace
diff --git a/src/test/modules/libpq_pipeline/traces/nosync.trace b/src/interfaces/libpq/test/traces/nosync.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/nosync.trace
rename to src/interfaces/libpq/test/traces/nosync.trace
diff --git a/src/test/modules/libpq_pipeline/traces/pipeline_abort.trace b/src/interfaces/libpq/test/traces/pipeline_abort.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/pipeline_abort.trace
rename to src/interfaces/libpq/test/traces/pipeline_abort.trace
diff --git a/src/test/modules/libpq_pipeline/traces/pipeline_idle.trace b/src/interfaces/libpq/test/traces/pipeline_idle.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/pipeline_idle.trace
rename to src/interfaces/libpq/test/traces/pipeline_idle.trace
diff --git a/src/test/modules/libpq_pipeline/traces/prepared.trace b/src/interfaces/libpq/test/traces/prepared.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/prepared.trace
rename to src/interfaces/libpq/test/traces/prepared.trace
diff --git a/src/test/modules/libpq_pipeline/traces/simple_pipeline.trace b/src/interfaces/libpq/test/traces/simple_pipeline.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/simple_pipeline.trace
rename to src/interfaces/libpq/test/traces/simple_pipeline.trace
diff --git a/src/test/modules/libpq_pipeline/traces/singlerow.trace b/src/interfaces/libpq/test/traces/singlerow.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/singlerow.trace
rename to src/interfaces/libpq/test/traces/singlerow.trace
diff --git a/src/test/modules/libpq_pipeline/traces/transaction.trace b/src/interfaces/libpq/test/traces/transaction.trace
similarity index 100%
rename from src/test/modules/libpq_pipeline/traces/transaction.trace
rename to src/interfaces/libpq/test/traces/transaction.trace
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9090226daa0..d3d18ab0d4a 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,7 +10,6 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
- libpq_pipeline \
plsample \
snapshot_too_old \
spgist_name_ops \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
deleted file mode 100644
index 3a11e786b83..00000000000
--- a/src/test/modules/libpq_pipeline/.gitignore
+++ /dev/null
@@ -1,5 +0,0 @@
-# Generated subdirectories
-/log/
-/results/
-/tmp_check/
-/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
deleted file mode 100644
index 65acc3e997e..00000000000
--- a/src/test/modules/libpq_pipeline/Makefile
+++ /dev/null
@@ -1,25 +0,0 @@
-# src/test/modules/libpq_pipeline/Makefile
-
-PGFILEDESC = "libpq_pipeline - test program for pipeline execution"
-PGAPPICON = win32
-
-PROGRAM = libpq_pipeline
-OBJS = $(WIN32RES) libpq_pipeline.o
-
-NO_INSTALL = 1
-
-PG_CPPFLAGS = -I$(libpq_srcdir)
-PG_LIBS_INTERNAL += $(libpq_pgport)
-
-TAP_TESTS = 1
-
-ifdef USE_PGXS
-PG_CONFIG = pg_config
-PGXS := $(shell $(PG_CONFIG) --pgxs)
-include $(PGXS)
-else
-subdir = src/test/modules/libpq_pipeline
-top_builddir = ../../../..
-include $(top_builddir)/src/Makefile.global
-include $(top_srcdir)/contrib/contrib-global.mk
-endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
deleted file mode 100644
index d8174dd579a..00000000000
--- a/src/test/modules/libpq_pipeline/README
+++ /dev/null
@@ -1 +0,0 @@
-Test programs and libraries for libpq
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 1de77075724..1a3064e9ab4 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -37,13 +37,14 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = {};
my @contrib_uselibpq = ();
-my @contrib_uselibpgport = ('libpq_uri_regress', 'libpq_testclient');
-my @contrib_uselibpgcommon = ();
+my @contrib_uselibpgport = ('libpq_uri_regress', 'libpq_testclient', 'libpq_pipeline');
+my @contrib_uselibpgcommon = ('libpq_pipeline');
my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = {};
my $contrib_extrasource = {
'libpq_uri_regress' => ['src/interfaces/libpq/test/libpq_uri_regress.c'],
'libpq_testclient' => ['src/interfaces/libpq/test/libpq_testclient.c'],
+ 'libpq_pipeline' => ['src/interfaces/libpq/test/libpq_pipeline.c'],
};
my @contrib_excludes = (
'bool_plperl', 'commit_ts',
--
2.17.1
>From ea665ea840756696b53232217f20cea28b3fe87e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 1 May 2022 07:55:57 -0500
Subject: [PATCH 16/21] msvc: do not install libpq test tools by default
See also:
https://www.postgresql.org/message-id/flat/20220501080706.GA1542365%40rfd.leadboat.com
a17fd67d2f2861ae0ce00d1aeefdf2facc47cd5e Build libpq test programs under MSVC.
https://www.postgresql.org/message-id/74952229-b3b0-fe47-f958-4088529a3...@dunslane.net MSVC build system installs extra executables
https://www.postgresql.org/message-id/e4233934-98a6-6f76-46a0-992c0f4f1...@dunslane.net Re: set TESTDIR from perl rather than Makefile
ci-os-only: linux, windows
---
.cirrus.yml | 1 +
src/tools/msvc/Install.pm | 13 +++++++++++++
2 files changed, 14 insertions(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index 1d16594803f..2e1290bc4a4 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -445,6 +445,7 @@ task:
tempinstall_script:
# Installation on windows currently only completely works from src/tools/msvc
+ - set INSTALL_TESTS=1
- cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install
this_tap_script: |
diff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm
index 8de79c618cb..f70dbb1daf2 100644
--- a/src/tools/msvc/Install.pm
+++ b/src/tools/msvc/Install.pm
@@ -29,6 +29,10 @@ my @client_program_files = (
'pg_config', 'pg_dump', 'pg_dumpall', 'pg_isready',
'pg_receivewal', 'pg_recvlogical', 'pg_restore', 'psql',
'reindexdb', 'vacuumdb', @client_contribs);
+my @test_program_files = (
+ 'isolationtester', 'libpq_pipeline', 'libpq_testclient',
+ 'libpq_uri_regress', 'pg_isolation_regress', 'pg_regress_ecpg',
+ 'pg_regress', 'zic');
sub lcopy
{
@@ -264,10 +268,19 @@ sub CopySolutionOutput
$sln =~ s/$rem//;
+ # Only install client tools
next
if ($insttype eq "client" && !grep { $_ eq $pf }
@client_program_files);
+ # Install test tools only in test mode
+ if (!$ENV{INSTALL_TESTS} && grep { $_ eq $pf }
+ @test_program_files)
+ {
+ print "Skipping install: $pf\n";
+ next;
+ }
+
my $proj = read_file("$pf.$vcproj")
|| croak "Could not open $pf.$vcproj\n";
--
2.17.1
>From defdf9503c7dd236f1cc12236ab9045092d32479 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Mon, 17 Jan 2022 00:54:28 -0600
Subject: [PATCH 17/21] cirrus: code coverage
https://www.postgresql.org/message-id/202202111821.w3gqblvfp4pr%40alvherre.pgsql
https://www.postgresql.org/message-id/flat/20220409021853.gp24...@telsasoft.com
ci-os-only: linux
---
.cirrus.yml | 16 ++++++++++++++++
src/tools/ci/code-coverage-report | 29 +++++++++++++++++++++++++++++
2 files changed, 45 insertions(+)
create mode 100755 src/tools/ci/code-coverage-report
diff --git a/.cirrus.yml b/.cirrus.yml
index 2e1290bc4a4..e09dbfe3857 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -28,6 +28,13 @@ env:
TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf
PG_TEST_EXTRA: kerberos ldap ssl
+ # The commit that this branch is rebased on. There's no easy way to find this.
+ # This does the right thing for cfbot, which always squishes all patches into a single commit.
+ # And does the right thing for any 1-patch commits.
+ # Patch series manually submitted to cirrus would benefit from setting this to the
+ # number of patches in the series (or directly to the commit the series was rebased on).
+ BASE_COMMIT: HEAD~1
+
# What files to preserve in case tests fail
on_failure: &on_failure
@@ -183,6 +190,7 @@ task:
cat /proc/cmdline
ulimit -a -H && ulimit -a -S
export
+ git diff --name-only "$BASE_COMMIT"
create_user_script: |
useradd -m postgres
chown -R postgres:postgres .
@@ -202,6 +210,7 @@ task:
su postgres <<-EOF
./configure \
--enable-cassert --enable-debug --enable-tap-tests \
+ --enable-coverage \
--enable-nls \
\
${LINUX_CONFIGURE_FEATURES} \
@@ -225,6 +234,13 @@ task:
make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS}
EOF
+ # Build coverage report for files changed since the base commit.
+ generate_coverage_report_script: |
+ src/tools/ci/code-coverage-report "$BASE_COMMIT"
+
+ coverage_artifacts:
+ paths: ['coverage/**/*.html', 'coverage/**/*.png', 'coverage/**/*.gcov', 'coverage/**/*.css' ]
+
on_failure:
<<: *on_failure
cores_script: src/tools/ci/cores_backtrace.sh linux /tmp/cores
diff --git a/src/tools/ci/code-coverage-report b/src/tools/ci/code-coverage-report
new file mode 100755
index 00000000000..0dce149dcf8
--- /dev/null
+++ b/src/tools/ci/code-coverage-report
@@ -0,0 +1,29 @@
+#! /bin/sh
+# Called during the linux CI task to generate a code coverage report.
+set -e
+
+base_branch=$1
+changed=`git diff --name-only "$base_branch" '*.c'` ||
+ [ $? -eq 1 ]
+
+outdir=coverage
+mkdir "$outdir"
+
+# Coverage is shown only for changed files
+# This is useful to see coverage of newly-added code, but won't
+# show added/lost coverage in files which this patch doesn't modify.
+
+gcov=$outdir/coverage.gcov
+for f in $changed
+do
+ # Avoid removed files
+ [ -f "$f" ] || continue
+
+ lcov --quiet --capture --directory "$f"
+done >"$gcov"
+
+# Exit successfully if no relevant files were changed
+[ -s "$gcov" ] || exit 0
+
+genhtml "$gcov" --show-details --legend --quiet --num-spaces=4 --output-directory "$outdir" --title="Coverage report of files changed since: $base_branch"
+cp "$outdir"/index.html "$outdir"/00-index.html
--
2.17.1
>From cf73892131beb1899aa91c255d9d4ba8e14343bc Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 26 Feb 2022 19:34:35 -0600
Subject: [PATCH 18/21] cirrus: build docs as a separate task..
Always run doc build; to allow them to be shown in cfbot.
A failure in documentation should be shown even if Linux fails check-world.
This'll automatically show up as a separate "column" on cfbot.
Also, in the future, this will hopefully upload each patch's changed HTML docs
as an artifact, for easy review.
XX-os-only: html
---
.cirrus.yml | 46 ++++++++++++++++++++++++++++++++--------------
1 file changed, 32 insertions(+), 14 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index e09dbfe3857..2ad21affb75 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -626,20 +626,6 @@ task:
time make -s -j${BUILD_JOBS} world-bin
ccache --show-stats
- ###
- # Verify docs can be built
- ###
- # XXX: Only do this if there have been changes in doc/ since last build
- always:
- docs_build_script: |
- time ./configure \
- --cache gcc.cache \
- CC="ccache gcc" \
- CXX="ccache g++" \
- CLANG="ccache clang"
- make -s -j${BUILD_JOBS} clean
- time make -s -j${BUILD_JOBS} -C doc
-
###
# Verify headerscheck / cpluspluscheck succeed
#
@@ -665,3 +651,35 @@ task:
always:
upload_caches: ccache
+
+
+###
+# Verify docs can be built
+###
+
+task:
+ name: Documentation
+
+ env:
+ CPUS: 1
+ BUILD_JOBS: 1
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
+ skip: "!changesInclude('.cirrus.yml', 'doc/**')"
+
+ container:
+ image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
+ cpu: $CPUS
+ memory: 2G
+
+ sysinfo_script: |
+ id
+ uname -a
+ cat /proc/cmdline
+ ulimit -a -H && ulimit -a -S
+ export
+
+ # Exercise HTML and other docs:
+ docs_build_script: |
+ time ./configure
+ make -s -j${BUILD_JOBS} -C doc
--
2.17.1
>From 931f1f7459f390140dacb82ef061d9abd184160a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 26 Feb 2022 19:39:10 -0600
Subject: [PATCH 19/21] cirrus: upload changed html docs as artifacts
This could be done on the client side (cfbot). One advantage of doing it here
is that fewer docs are uploaded - many patches won't upload docs at all.
https://www.postgresql.org/message-id/flat/20220409021853.gp24...@telsasoft.com
https://cirrus-ci.com/task/5396696388599808
ci-os-only: html
---
.cirrus.yml | 19 +++++++++++++++++--
src/tools/ci/copy-changed-docs | 23 +++++++++++++++++++++++
2 files changed, 40 insertions(+), 2 deletions(-)
create mode 100755 src/tools/ci/copy-changed-docs
diff --git a/.cirrus.yml b/.cirrus.yml
index 2ad21affb75..53be4f6f4b6 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -654,7 +654,7 @@ task:
###
-# Verify docs can be built
+# Verify docs can be built, and (only on cfbot) upload changed docs as artifacts
###
task:
@@ -665,7 +665,7 @@ task:
BUILD_JOBS: 1
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
- skip: "!changesInclude('.cirrus.yml', 'doc/**')"
+ #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')"
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
@@ -678,8 +678,23 @@ task:
cat /proc/cmdline
ulimit -a -H && ulimit -a -S
export
+ git diff --name-only "$BASE_COMMIT"
# Exercise HTML and other docs:
docs_build_script: |
time ./configure
make -s -j${BUILD_JOBS} -C doc
+ cp -r doc new-docs
+
+ # Build HTML docs from the base commit.
+ git checkout "$BASE_COMMIT" -- doc
+ #html-stamp
+ make -s -C doc clean
+ make -s -C doc html
+ cp -r doc old-docs
+
+ copy_changed_docs_script: |
+ src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
+
+ html_docs_artifacts:
+ paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
new file mode 100755
index 00000000000..1584f974d94
--- /dev/null
+++ b/src/tools/ci/copy-changed-docs
@@ -0,0 +1,23 @@
+#! /bin/sh
+# Copy HTML which differ between $old and $new into $outdir
+set -e
+
+old=$1
+new=$2
+outdir=$3
+
+mkdir "$outdir"
+cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+
+changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
+ [ $? -eq 1 ]
+
+for f in $changed
+do
+ # Avoid removed files
+ [ -f "$f" ] || continue
+
+ cp -v "$f" "$outdir"
+done
+
+exit 0
--
2.17.1
>From 79ea3803d7befc2ee6b22355113d5e0bc22ce9af Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Mon, 28 Feb 2022 23:18:19 -0600
Subject: [PATCH 20/21] f!html: index file
This allows linking to the artifacts from the last successful build, which
itself allows *not* rebuilding when sources haven't changed.
ci/os/only: html
---
.cirrus.yml | 4 ++--
src/tools/ci/copy-changed-docs | 16 ++++++++++++++--
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 53be4f6f4b6..2b7664485f6 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -33,7 +33,7 @@ env:
# And does the right thing for any 1-patch commits.
# Patch series manually submitted to cirrus would benefit from setting this to the
# number of patches in the series (or directly to the commit the series was rebased on).
- BASE_COMMIT: HEAD~1
+ BASE_COMMIT: HEAD~2
# What files to preserve in case tests fail
@@ -694,7 +694,7 @@ task:
cp -r doc old-docs
copy_changed_docs_script: |
- src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs"
+ src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs" "$CIRRUS_BRANCH"
html_docs_artifacts:
paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css']
diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs
index 1584f974d94..b0a4c971aaf 100755
--- a/src/tools/ci/copy-changed-docs
+++ b/src/tools/ci/copy-changed-docs
@@ -5,10 +5,16 @@ set -e
old=$1
new=$2
outdir=$3
+branch=$4
mkdir "$outdir"
cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir"
+# The index is useful to allow a static link to the artifacts for the most-recent, successful CI run for a branch
+# https://api.cirrus-ci.com/v1/artifact/github/USERNAME/postgres/Documentation/html_docs/html_docs/00-doc.html?branch=BRANCH
+index="$outdir/00-doc.html"
+echo "<html><head><title>Index of docs changed since: $branch</title></head><body><ul>" >"$index"
+
changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` ||
[ $? -eq 1 ]
@@ -17,7 +23,13 @@ do
# Avoid removed files
[ -f "$f" ] || continue
- cp -v "$f" "$outdir"
-done
+ cp -v "$f" "$outdir" >&2
+ fn=${f##*/}
+ # ?branch=... is needed for the static link for the branch
+ # It's not used if accessing artifacts for *this* CI run
+ echo "<li><a href='$fn?branch=$branch'>$fn</a>"
+done >>"$index"
+
+echo "</ul></body></html>" >>"$index"
exit 0
--
2.17.1
>From ddb306f95366059d9f8e4218d561e2d96696ee38 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 24 Jun 2022 00:09:12 -0500
Subject: [PATCH 21/21] cirrus: run freebsd with more CPUs+RAM
This reduces test time from 10min to 3min.
4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720
4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008
4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600
6 CPUs https://cirrus-ci.com/task/6678321684545536
8 CPUs https://cirrus-ci.com/task/6264854121021440
See also:
https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4h...@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686
8 jobs 7min https://cirrus-ci.com/task/6186376667332608
xi-os-only: freebsd
---
.cirrus.yml | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 2b7664485f6..6e799cc764e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -49,11 +49,9 @@ task:
name: FreeBSD - 13
env:
- # FreeBSD on GCP is slow when running with larger number of CPUS /
- # jobs. Using one more job than cpus seems to work best.
- CPUS: 2
- BUILD_JOBS: 3
- TEST_JOBS: 3
+ CPUS: 4
+ BUILD_JOBS: 4
+ TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
@@ -64,7 +62,7 @@ task:
image: family/pg-ci-freebsd-13-0
platform: freebsd
cpu: $CPUS
- memory: 2G
+ memory: 3G
disk: 50
sysinfo_script: |
--
2.17.1
