Re: Adding CI to our tree (ccache)

2022-02-20 Thread Andres Freund
easily solvable by using a different debug format IIRC (/Z7 or such). Greetings, Andres Freund

Re: initdb / bootstrap design

2022-02-20 Thread Andres Freund
s.bki processing would make invalidation handling for subsequent indexes faster. Or maybe we can disable a few more invalidations. Inval processing is >10% - more than 10% (assert) / 7% (optimized) is spent in compute_scalar_stats()->qsort_arg(). Something seems off with that to me. C

Re: do only critical work during single-user vacuum?

2022-02-20 Thread Andres Freund
f FreezeMultiXactId() needs to create a new multixact for a far-in-the-past xid. That's not great, of course, but if we include the precise cause (pid of backend / prepared xact name / slot name / ...) necessitating creating a new multi, it'd still be a significant improvement over the status quo. Greetings, Andres Freund

Re: set TESTDIR from perl rather than Makefile

2022-02-20 Thread Andres Freund
ir/$Config/pg_regress/pg_regress"; > $ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll"; > > - $ENV{TESTDIR} = "$dir"; > my $module = basename $dir; > - # add the module build dir as the second element in the PATH > - $ENV{PATH} =~ s!;!;$topdir/$Config/$module;!; > + #$ENV{VCREGRESS_MODE} = $Config; Hm. How does the module build dir end up on PATH after this? Greetings, Andres Freund

Re: pg_upgrade verbosity when redirecting output to log file

2022-02-20 Thread Andres Freund
ss the comment should say "or in verbose mode". Indeed. I think I got caught in a back-and-forth between different formulations. Baring that, anybody against committing this? Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Andres Freund
omewhere in pgstats. But it shouldn't be split by worker or relation, and it shouldn't contain non-cumulative error information. Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Andres Freund
have pg_stat_ stuff that's not actually stats, but something describing the current state, but that ship has well sailed). Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-21 Thread Andres Freund
Hi, On 2022-02-21 14:49:01 +0530, Amit Kapila wrote: > On Mon, Feb 21, 2022 at 1:18 PM Andres Freund wrote: > > > * stats_reset (Time at which these statistics were last reset) > > > > > > The view name could be pg_stat_subscription_lrep, > > > pg_stat_log

Re: pg_upgrade verbosity when redirecting output to log file

2022-02-21 Thread Andres Freund
On 2022-02-21 15:29:09 +0100, Daniel Gustafsson wrote: > > On 21 Feb 2022, at 02:07, Andres Freund wrote: > > > Baring that, anybody against committing this? > > LGTM. The above mentioned comment was the only thing I found as well. Thanks for the review Justin and Daniel. Pushed.

making pg_regress less noisy by removing boilerplate

2022-02-21 Thread Andres Freund
from that. But if I understood that patch, it'd also benefit from the reduction in unnecessary headers, albeit not as crucially. Greetings, Andres Freund [1] https://postgr.es/m/62A6B9F0-C3D3-45CD-8E8B-90A8E5B08DFA%40yesql.se

Re: [RFC] building postgres with meson

2022-02-21 Thread Andres Freund
gt; {"config-auth", required_argument, NULL, 24}, > {"max-concurrent-tests", required_argument, NULL, 25}, > {"make-testtablespace-dir", no_argument, NULL, 26}, > + {"tap", no_argument, NULL, 27}, &g

Re: making pg_regress less noisy by removing boilerplate

2022-02-21 Thread Andres Freund
all to create the dependencies, and a prefix # rule for the recipe. # # @ silences, : is the shell do-nothing command. %.silence: @: define create_silence_target $(1): | $(1).silence endef $(foreach t,$(standard_targets) $(standard_always_targets) generated-header-symlinks,$(eval $(call create_silence_target,$(t Greetings, Andres Freund

Re: Time to drop plpython2?

2022-02-21 Thread Andres Freund
ful to run out-of-support distribution versions. Leaving security etc aside, not being able to install packages seems sufficient reason to upgrade. Greetings, Andres Freund [1] https://lists.opensuse.org/archives/list/security-annou...@lists.opensuse.org/message/6CIQPV3H6J4AIIXUKUGI4IESMTHIFFFB/

Re: making pg_regress less noisy by removing boilerplate

2022-02-21 Thread Andres Freund
Hi, On 2022-02-21 12:40:18 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-02-21 12:05:42 -0500, Tom Lane wrote: > >> except maybe the "running on port 51696 with PID 1156405" line (and I'm not > >> too wedded to that)? > > > We stil

Re: External data files possible?

2022-02-21 Thread Andres Freund
7;t think there's a way around that right now. > Is there a hook I can use to clean these files up? More generally, can I > get away with using my own data files without causing a problem? Not currently. A plain hook wouldn't suffice, because it'd not integrate with transactional DDL and crash recovery. Greetings, Andres Freund

Re: speed up a logical replica setup

2022-02-21 Thread Andres Freund
e to allow to call pg_basebackup from the new tool. Perhaps with a --pg-basebackup-parameters or such. Greetings, Andres Freund

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-02-22 Thread Andres Freund
Hi, On 2022-02-10 14:26:59 -0800, Andres Freund wrote: > On 2022-02-11 09:10:38 +1300, Thomas Munro wrote: > > It seems like I should go ahead and do that today, and we can study > > further uses for PROCSIGNAL_BARRIER_SMGRRELEASE in follow-on work? > > Yes. I wrote a tes

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-02-22 Thread Andres Freund
Hi, On 2022-02-22 01:11:21 -0800, Andres Freund wrote: > I've started to work on a few debugging aids to find problem like > these. Attached are two WIP patches: Forgot to attach. Also importantly includes a tap test for several of these issues Greetings, Andres F

Re: TAP output format in pg_regress

2022-02-22 Thread Andres Freund
'd much rather not use BAIL - I haven't gotten around to doing anything about it, but I really want to get rid of nearly all our uses of bail: https://www.postgresql.org/message-id/20220213232249.7sevhlioapydla37%40alap3.anarazel.de Greetings, Andres Freund

Re: bailing out in tap tests nearly always a bad idea

2022-02-22 Thread Andres Freund
Hi, On 2022-02-14 09:46:20 -0800, Andres Freund wrote: > > t/die-immediately.t aieee at t/die-immediately.t line 1. > > t/die-immediately.t Dubious, test returned 255 (wstat 65280, 0xff00) > > No subtests run > > Hm. Looks different when a test is including

Re: bailing out in tap tests nearly always a bad idea

2022-02-22 Thread Andres Freund
Hi, On 2022-02-22 09:28:37 -0800, Andres Freund wrote: > On 2022-02-14 09:46:20 -0800, Andres Freund wrote: > > > t/die-immediately.t aieee at t/die-immediately.t line 1. > > > t/die-immediately.t Dubious, test returned 255 (wstat 65280, 0xff00) > > > No

Re: bailing out in tap tests nearly always a bad idea

2022-02-22 Thread Andres Freund
On 2022-02-22 15:10:30 -0500, Andrew Dunstan wrote: > I'll be surprised if we can't come up with something cleaner than that. Suggestions?

Re: making pg_regress less noisy by removing boilerplate

2022-02-22 Thread Andres Freund
ests aren't run concurrently (whereas tap tests can be run concurrently with PROVE_FLAGS-j). I think we just make make_temp_sockdir() a tad smarter. Perhaps by lifting the code in src/bin/psql/command.c:do_edit() to src/port/path.c or such? Greetings, Andres Freund

Re: small development tip: Consider using the gold linker

2022-02-22 Thread Andres Freund
Hi, On 2022-01-13 19:24:09 -0800, Peter Geoghegan wrote: > On Fri, Jul 20, 2018 at 8:16 PM Peter Geoghegan wrote: > > On Mon, Jul 9, 2018 at 4:40 PM, Andres Freund wrote: > > > FWIW, I've lately noticed that I spend a fair time waiting for the > > > linker durin

Re: Race condition in InvalidateObsoleteReplicationSlots()

2022-02-22 Thread Andres Freund
pareToSleep(&s->active_cv); LWLockRelease(ReplicationSlotControlLock); but afaics that is bogus, because releasing a slot doesn't take ReplicationSlotControlLock. That just protects against the slot being dropped, not against it being released. We can Condi

Re: Race condition in InvalidateObsoleteReplicationSlots()

2022-02-22 Thread Andres Freund
Hi, On 2022-02-22 17:48:55 -0800, Andres Freund wrote: > I think the minor changes might unfortunately have introduced a race? Before > the patch just used ConditionVariableSleep(), but now it also has a > ConditionVariablePrepareToSleep(). Without re-checking the sleep conditio

Re: Race conditions in 019_replslot_limit.pl

2022-02-22 Thread Andres Freund
ogs of primary3 don't have a single mention of ereport(LOG, (errmsg("terminating process %d to release replication slot \"%s\"", active_pid, NameStr(slotname)))); On 2022-02-18 15:14:15 -0800, Andres Freu

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Andres Freund
uldn't start a worker because the max numbers of workers was already active or such. Greetings, Andres Freund

Re: Use generation context to speed up tuplesorts

2022-02-22 Thread Andres Freund
e a win even in less adversarial situations than "a single tuple per generation block". Greetings, Andres Freund

Re: catalog access with reset GUCs during parallel worker startup

2022-02-22 Thread Andres Freund
parallel_mode = on; SET postgres[3312622][1]=> SELECT current_setting('session_authorization'); ┌─┐ │ current_setting │ ├─────┤ │ andres │ └─┘ (1 row) but that could be remedied just already done for role. Greetings, Andres Freund

Re: Frontend error logging style

2022-02-22 Thread Andres Freund
os in favor of using pg_log_fatal > directly; but I've not done so here, since this patch is eyewateringly > long and boring already. Agreed. I don't care that much about the naming here. For a moment I was wondering whether there'd be a benefit for having the "pure logging" stuff be in a different static library than the erroring variant. So that e.g. libpq's check for exit() doesn't run into trouble. But then I remembered that libpq doesn't link against common... Greetings, Andres Freund

Re: Frontend error logging style

2022-02-22 Thread Andres Freund
On 2022-02-22 22:44:25 -0500, Tom Lane wrote: > Andres Freund writes: > > What about adding a pg_fatal() that's pg_log_fatal() + exit()? That keeps > > pg_log_* stuff "log only", but adds something adjacent enough to hopefully > > reduce future misunders

Re: bailing out in tap tests nearly always a bad idea

2022-02-23 Thread Andres Freund
Hi, On 2022-02-23 08:43:38 -0500, Andrew Dunstan wrote: > > On 2/22/22 15:54, Andres Freund wrote: > > On 2022-02-22 15:10:30 -0500, Andrew Dunstan wrote: > >> I'll be surprised if we can't come up with something cleaner than that. > > Suggestions? > >

Re: Frontend error logging style

2022-02-23 Thread Andres Freund
y-in that this solution would be > okay to commit. Any objections? +1 Greetings, Andres Freund

convert libpq uri-regress tests to tap test

2022-02-23 Thread Andres Freund
e src/test/modules/libpq_pipeline to src/test/modules/libpq and move uri-regress in there as well? The tap test needs a bit more polish, mostly posted this to get some feedback before wasting more time :) Greetings, Andres Freund >From 77b3666c9a7f17f98120cfc9c2a8e99f7d4ab491 Mon Sep 17 00:00

Re: convert libpq uri-regress tests to tap test

2022-02-23 Thread Andres Freund
Hi, On 2022-02-23 15:57:08 -0500, Tom Lane wrote: > Andrew Dunstan writes: > > On 2/23/22 15:30, Andres Freund wrote: > >> Perhaps we should just rename src/test/modules/libpq_pipeline to > >> src/test/modules/libpq and move uri-regress in there as well? > > &g

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Andres Freund
> Earlier, pg_stat_subscription_stats was mentioned, which doesn't have that > problem. We really should try to fix this in a more general way at some point. We have way too many different things mixed up in pg_stat_*. I'd like to get something like the patch in soon though, we can still change the name later. I've been blocked behind this stuff for weeks, and it's getting really painful... Greetings, Andres Freund

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
ot > that important. Why not in t/? We can't easily build the test programs in libpq/ itself, but libpq/t should be fairly doable. Greetings, Andres Freund

Re: Frontend error logging style

2022-02-24 Thread Andres Freund
anything by not standardizing on one notion of a fatal error wrapper. Greetings, Andres Freund

Re: Readd use of TAP subtests

2022-02-24 Thread Andres Freund
# Subtest: vacuumlo --help > ok 1 - exit code 0 > ok 2 - goes to stdout > ok 3 - nothing to stderr > 1..3 It's clearly better. We can approximate some of it by using is_deeply() and comparing exit, stdout, stderr at once. Particularly for helpers like program_help() that are used in a lot of places. Greetings, Andres Freund

Re: create_index test fails when synchronous_commit = off @ master

2022-02-24 Thread Andres Freund
options and where the effort required is reasonable. And some cases where it's not... synchronous_commit=off worked until recently, and I think we should keep it working. Greetings, Andres Freund

Re: create_index test fails when synchronous_commit = off @ master

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 07:33:39 -0800, Andres Freund wrote: > I added the SELECT relpages, reltuples, relallvisible FROM pg_class WHERE oid > = 'tenk1'::regclass; > just after the > VACUUM ANALYZE tenk1; > > synchronous_commit=on > + relpa

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 10:17:28 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-02-24 13:31:40 +0100, Peter Eisentraut wrote: > >> I think the TAP scripts should be in src/interfaces/libpq/t/, as usual. The > >> supporting code snippets could live in some oth

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 17:03:33 +, Jacob Champion wrote: > On Thu, 2022-02-24 at 08:46 -0800, Andres Freund wrote: > > One annoying bit is that our current tap invocation infrastructure for msvc > > won't know how to deal with that. We put the build directory containing t/ >

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Andres Freund
ich is a lot of people) will be fine without a rebuild. And it also > makes the indexes a bit smaller, thanks to saving 20B. Won't 2) also break indexes created without a pg_upgrade? "already upgraded from 12" sounds like it wouldn't but I don't see why? Greetings, Andres Freund

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Andres Freund
Hi, On February 24, 2022 5:44:57 PM PST, Tom Lane wrote: >Tomas Vondra writes: >> I wonder if we could check the index tuple length, and adjust the siglen >> based on that, somehow ... > >In an already-corrupted index, there won't be a unique answer. If we're breaking every ltree gist index, c

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread Andres Freund
g that that's easy, but I don't think it's a fundamental restriction. I also can imagine things like using selinux labeling of connections. We have several useful authentication technologies built ontop of plaintext exchange. Radius, Ldap, Pam afaics could be implemented as an extension? Greetings, Andres Freund

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-24 Thread Andres Freund
y is a backend running, what user, etc). They're not unrelated though: backend_status.c feeds pgstat.c with some information occasionally. Greetings, Andres Freund

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-24 Thread Andres Freund
of skippable pages. > There is no chance of VACUUM advancing relfrozenxid in this scenario > either way, though, so it doesn't matter. I think this commit message needs a good amount of polishing - it's very convoluted. It's late and I didn't sleep well, but I've tried to read it several times without really getting a sense of what this precisely does. > From 15dec1e572ac4da0540251253c3c219eadf46a83 Mon Sep 17 00:00:00 2001 > From: Peter Geoghegan > Date: Thu, 24 Feb 2022 17:21:45 -0800 > Subject: [PATCH v9 4/4] Avoid setting a page all-visible but not all-frozen. To me the commit message body doesn't actually describe what this is doing... > This is pretty much an addendum to the work in the "Make page-level > characteristics drive freezing" commit. It has been broken out like > this because I'm not even sure if it's necessary. It seems like we > might want to be paranoid about losing out on the chance to advance > relfrozenxid in non-aggressive VACUUMs, though. > The only test that will trigger this case is the "freeze-the-dead" > isolation test. It's incredibly narrow. On the other hand, why take a > chance? All it takes is one heap page that's all-visible (and not also > all-frozen) nestled between some all-frozen heap pages to lose out on > relfrozenxid advancement. The SKIP_PAGES_THRESHOLD stuff won't save us > then [1]. FWIW, I'd really like to get rid of SKIP_PAGES_THRESHOLD. It often ends up causing a lot of time doing IO that we never need, completely trashing all CPU caches, while not actually causing decent readaead IO from what I've seen. Greetings, Andres Freund

Re: [Proposal] Global temporary tables

2022-02-24 Thread Andres Freund
completely separate implementation for GTTs? That seems unmaintainable. > +/*- > + * > + * storage_gtt.c > + * The body implementation of Global temparary table. > + * > + * IDENTIFICATION > + * src/backend/catalog/storage_gtt.c > + * > + * See src/backend/catalog/GTT_README for Global temparary table's > + * requirements and design. > + * > + *- > + */ I don't think that path to the readme is correct. Greetings, Andres Freund

Re: BufferAlloc: don't take two simultaneous locks

2022-02-25 Thread Andres Freund
p; BM_LOCKED)) I don't think it's safe to atomic in arbitrary bits. If somebody else has locked the buffer header in this moment, it'll lead to completely bogus results, because unlocking overwrites concurrently written contents (which there shouldn't be any, but here there are)... And or'ing contents in also doesn't make sense because we it doesn't work to actually unset any contents? Why don't you just use LockBufHdr/UnlockBufHdr? Greetings, Andres Freund

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Andres Freund
k down those cases where JIT ends up taking 90% of the > query runtime because of bad estimates... Hm. Could it make sense to do this as a auto_explain feature? Greetings, Andres Freund

Re: Readd use of TAP subtests

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:39:15 +0100, Peter Eisentraut wrote: > On 24.02.22 16:00, Andres Freund wrote: > > I've incidentally played with subtests yesterdays, when porting > > src/interfaces/libpq/test/regress.pl to a tap test. Unfortunately it seems > > that subtests aren&#

Re: Readd use of TAP subtests

2022-02-25 Thread Andres Freund
t in the future. TAP::Harness reports TAP syntax errors at the end of a test run. " I honestly don't know what to make of that. Parsers are supposed to ignore unknown lines, Test::Harness silently ignores, but also "TAP::Harness reports TAP syntax errors"? This seems like a self contradictory mess. Greetings, Andres Freund

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 17:28:41 +0100, Magnus Hagander wrote: > On Fri, Feb 25, 2022 at 5:20 PM Andres Freund wrote: > > On 2022-02-25 16:16:01 +0100, Magnus Hagander wrote: > > > This patch adds a configuration parameter jit_warn_above_fraction that > > > will cause a

Re: BufferAlloc: don't take two simultaneous locks

2022-02-25 Thread Andres Freund
Sorry, I didn't understand sentence :(( You're OR'ing multiple bits into buf->state. LockBufHdr() only ORs in BM_LOCKED. ORing BM_LOCKED is fine: Either the buffer is not already locked, in which case it just sets the BM_LOCKED bit, acquiring the lock. Or it doesn't change anyt

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Andres Freund
net negative for security. Not sure that I think > > it's a good idea. > > I don't understand your point. Can't you just use "hostssl" rather than > "host"? And the extension could check Port->ssl_in_use before sendAuthRequest(AUTH_REQ_PASSWORD) if it wanted to restrict it. Greetings, Andres Freund

Re: convert libpq uri-regress tests to tap test

2022-02-25 Thread Andres Freund
Hi, On 2022-02-24 08:46:23 -0800, Andres Freund wrote: > I'm mildly inclined to only do 0001 and 0002 for now. We'd not loose msvc > coverage, because it already doesn't build the test. Once we've ironed that > stuff out, we could do 0003? >From what I can see i

Re: Two noncritical bugs of pg_waldump

2022-02-25 Thread Andres Freund
rrno before the call to read(). > > So I post a patch contains only the indisputable part. Thanks for the report and fix. Pushed. This was surprisingly painful, all but one branch had conflicts... Greetings, Andres Freund

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-22 18:06:24 -0800, Andres Freund wrote: > On 2022-02-18 15:14:15 -0800, Andres Freund wrote: > > I'm running out of ideas for how to try to reproduce this. I think we might > > need some additional debugging information to get more information from the >

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 16:25:01 -0300, Euler Taveira wrote: > On Fri, Feb 25, 2022, at 11:52 AM, Greg Stark wrote: > > On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > > > > > Hi, > > > > > > I was looking the shared memory stats patch again. >

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Andres Freund
eds to do something more complicated than CheckSASLAuth(), it can use AUTH_REQ_SASL{,_FIN CONT} itself. Of course whether it's viable depends on what the custom auth method wants to do. But it's not a restriction of the authentication plugin model. Greetings, Andres Freund

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 15:07:01 -0500, Tom Lane wrote: > Andres Freund writes: > > Seems to suggest something is holding a problematic lock in a way that I do > > not understand yet: > > > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2022-02-23

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Andres Freund
when executed as part of a parallel query. I don't think we should add further functions not prefixed with pg_. Perhaps a few tests for less trivial authn_ids could be worthwhile? E.g. certificate DNs. Greetings, Andres Freund

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Andres Freund
plexity. Suddenly there's a bunch more initialization order dependencies that you don't have right now. I'd not go there for just this. Greetings, Andres Freund

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:00:12 -0800, Peter Geoghegan wrote: > On Thu, Feb 24, 2022 at 11:14 PM Andres Freund wrote: > > I am not a fan of the backstop terminology. It's still the reason we need to > > do freezing for correctness reasons. > > Thanks for the review! &

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Andres Freund
27;s the bound), and they have always more than one member. > How many of these "skewed MultiXacts" can we really expect? I don't think they're skewed in any way. It's a fundamental aspect of multixacts. Greetings, Andres Freund

Re: Mingw task for Cirrus CI

2022-02-25 Thread Andres Freund
g_config.h:875: note: this is the location of the previous definition [22:44:24.456] 875 | #define SIZEOF_OFF_T 4 Seems we're doing something wrong and end up with a 4 byte off_t, whereas python ends up with an 8byte one. We probably need to fix that. But it's not the cause of this problem. You could take out -s from the make flags and see whether plpython3.dll is being built and installed, and where to. Greetings, Andres Freund

Re: convert libpq uri-regress tests to tap test

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 09:56:47 -0800, Andres Freund wrote: > On 2022-02-24 08:46:23 -0800, Andres Freund wrote: > > I'm mildly inclined to only do 0001 and 0002 for now. We'd not loose msvc > > coverage, because it already doesn't build the test. Once we've ironed t

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 12:15:58 -0800, Andres Freund wrote: > On 2022-02-25 15:07:01 -0500, Tom Lane wrote: > > Andres Freund writes: > > Hmm. Maybe put a couple more debug messages into ReplicationSlotCleanup > > and/or ReplicationSlotDropPtr? It doesn't seem very clear

Re: Missed condition-variable wakeups on FreeBSD

2022-02-26 Thread Andres Freund
s=, nevents=) at latch.c:1171 The next time this happens / if you still have this open, perhaps it could be worth checking if there's a byte in the self pipe? > Thoughts? Ideas on debugging this? Besides trying to make the issue more likely as suggested above, it might be worth checking if signalling the stuck processes with SIGUSR1 gets them unstuck. Greetings, Andres Freund

Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-02-26 Thread Andres Freund
t this concrete problem for 14 and master, because it's certainly worse than before on linux / freebsd. I suspect the easiest is to just convert that usleep to a WaitLatch(). That'd require adding a new enum value to WaitEventTimeout in 14. Which probably is fine? Greetings, Andres Freund

Re: Fix a typo in pg_receivewal.c's get_destination_dir()

2022-02-26 Thread Andres Freund
harm in it as basedir is a global variable in > pg_receivewal.c, let's use the function parameter to be in sync with > close_destination_dir. > > Attaching a tiny patch to fix it. Good catch, pushed. Greetings, Andres Freund

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
s_script > > I'm not sure how to write this test in windows shell; it's also not easy to > write it in posix sh, since windows shell is somehow interpretting && and > ||... You could put the script in src/tools/ci and call it from the script to avoid the quoting issues. Would be good to add a comment explaining the fileLoggerParameters1 thing and a warning that compiler_warnings_script should be the last script. Greetings, Andres Freund

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
On 2022-02-26 17:09:08 -0800, Andres Freund wrote: > You could put the script in src/tools/ci and call it from the script to avoid > the quoting issues. Might also be a good idea for the bulk of the docs / coverage stuff, even if there are no quoting issues.

Re: convert libpq uri-regress tests to tap test

2022-02-26 Thread Andres Freund
Hi, On 2022-02-25 17:52:29 -0800, Andres Freund wrote: > I'd like to commit 0001 and 0002 soon, unless somebody sees a reason not to? Pushed. Attached is the remainder, 0003, the move of libpq_pipeline to src/interfaces/libpq that I'm not planning to push for now. Regards,

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
es each commit is independent from the prior commit, so it should rebuild it every time if the CF entry has changes to the docs. Greetings, Andres Freund

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
t replaces the prior commit, so I suspect it'll always be viewn as new. But who knows, shouldn't be hard to figure out. Greetings, Andres Freund

Re: [Proposal] Global temporary tables

2022-02-26 Thread Andres Freund
Without this, the GTT will be terribly slow like current temporary tables > with a lot of problems with bloating of pg_class, pg_attribute and > pg_depend tables. I think it's not a great idea to solve multiple complicated problems at once... Greetings, Andres Freund

Re: Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-02-27 Thread Andres Freund
Hi, On February 27, 2022 4:19:21 PM PST, Thomas Munro wrote: >With the attached, 027_stream_regress.pl drops from ~29.5s to ~19.6s >on my FreeBSD workstation! That's impressive - wouldn't have guessed it to make that much of a difference. I assume that running the tests on freebsd for an olde

Re: convert libpq uri-regress tests to tap test

2022-03-01 Thread Andres Freund
Hi, On March 1, 2022 7:44:27 AM PST, Andrew Dunstan wrote: > >On 2/24/22 07:19, Andrew Dunstan wrote: >> On 2/23/22 20:52, Tom Lane wrote: >>> Peter Eisentraut writes: On 23.02.22 23:58, Tom Lane wrote: > Peter Eisentraut writes: >> libpq TAP tests should be in src/interfaces/libp

Re: Proposal: Support custom authentication methods using hooks

2022-03-01 Thread Andres Freund
t seem flexible enough (e.g. no access to group memberships etc). Nor does baking stuff like that in seem realistic to me, it'll presumably be too cloud provider specific. Greetings, Andres Freund

Re: Proposal: Support custom authentication methods using hooks

2022-03-01 Thread Andres Freund
tom auth method can use SCRAM if relevant for it, with a small amount of code. So the whole plaintext discussion seems independent. Samay, what do you think about updating the test plugin to do SCRAM instead of plaintext, just to highlight that fact? Greetings, Andres Freund

Re: Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-03-01 Thread Andres Freund
ll_cv, > + > WAIT_EVENT_FORWARD_SYNC_REQUEST); > + LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE); > + } > + ConditionVariableCancelSleep(); > + } Could there be a problem with a lot of backends trying to acquire CheckpointerCommLock in exclusive mode? I'm inclined to think it's rare enough to not worry. Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-01 Thread Andres Freund
it in a more general way if we want it. Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-01 Thread Andres Freund
Hi, On 2022-03-02 12:39:57 +0530, Amit Kapila wrote: > On Wed, Mar 2, 2022 at 10:39 AM Andres Freund wrote: > > > > Working on rebasing shared memory stats over this. Feels *much* better so > > far. > > > > Good to hear that it helps. BTW, there is another patc

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-03-02 Thread Andres Freund
On 2022-02-15 13:02:41 +0900, Michael Paquier wrote: > >> + @regress_command = (@regress_command, @extra_opts); > >> + > >> + $oldnode->command_ok(@regress_command, > >> + 'regression test run on old instance'); > > > > I also think this should take EXTRA_REGRESS_OPTS into account - tes

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-03-02 Thread Andres Freund
nstall})) > + || (!defined($ENV{olddump}) && defined($ENV{oldinstall}))) Odd indentation. Spaces between parens? > +$newnode->init(extra => [ '--wal-segsize', '1', '--allow-group-access' ]); I'd copy the comments from test.sh wrt --wal-segsize, --allow-group-access. Greetings, Andres Freund

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-03-02 Thread Andres Freund
ot as a default part of the build. But ... maybe? Aggreed. I think it's racy anyway, because further files could get deleted (e.g. segments > 0) after the barrier has been processed. What I am stuck on is what we can do for the released branches. Data corruption after two consecutive ALTER DATABASE SET TABLESPACEs seems like something we need to address. Greetings, Andres Freund

Re: Proposal: Support custom authentication methods using hooks

2022-03-02 Thread Andres Freund
Hi, On 2022-03-02 09:32:26 +0100, Peter Eisentraut wrote: > On 01.03.22 22:34, Andres Freund wrote: > > The cases I've heard about are about centralizing auth across multiple cloud > > services. Including secret management in some form. E.g. allowing an > > application

Re: Proposal: Support custom authentication methods using hooks

2022-03-02 Thread Andres Freund
s good) that pass plaintext passwords > around. Shrug. I don't think it's a good idea to just leave people hanging without a replacement. It's OK to make it a bit harder and require explicit configuration, but dropping support for reasonable configurations IMO is something we should be very hesitant doing. Greetings, Andres Freund

Re: [Proposal] Global temporary tables

2022-03-02 Thread Andres Freund
Hi, On 2022-02-27 06:09:54 +0100, Pavel Stehule wrote: > ne 27. 2. 2022 v 5:13 odesílatel Andres Freund napsal: > > On 2022-02-27 04:17:52 +0100, Pavel Stehule wrote: > > > Without this, the GTT will be terribly slow like current temporary tables > > > with a lot o

Re: Add 64-bit XIDs into PostgreSQL 15

2022-03-02 Thread Andres Freund
ealistic target for this patch. There's huge amounts of work left, it has gotten very little review. I encourage trying to break down the patch into smaller incrementally useful pieces. E.g. making all the SLRUs 64bit would be a substantial and independently committable piece. Greetings, Andres Freund

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-02 Thread Andres Freund
n_id, opening up a security/monitoring hole or such because of a bogus return value. Greetings, Andres Freund

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-02 Thread Andres Freund
Hi, On 2022-02-25 11:32:24 -0800, Andres Freund wrote: > On 2022-02-25 16:25:01 -0300, Euler Taveira wrote: > > On Fri, Feb 25, 2022, at 11:52 AM, Greg Stark wrote: > > > On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > > > > > > > Hi, > > &

Re: casting operand to proper type in BlockIdGetBlockNumber

2022-03-03 Thread Andres Freund
nt,undefined -fno-sanitize=nonnull-attribute -fno-sanitize=float-cast-overflow -fno-sanitize-recover=all Need to manually add -ldl, because -fsanitize breaks our dl test (it uses dlopen, but not dlsym). Was planning to submit a fix for that... Greetings, Andres Freund

Re: casting operand to proper type in BlockIdGetBlockNumber

2022-03-03 Thread Andres Freund
Hi, On 2022-03-03 12:45:22 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-03-03 12:13:40 -0500, Tom Lane wrote: > >> Hmph. I tried enabling -fsanitize=undefined here, and I get some > >> complaints about passing null pointers to memcmp and the like, but &g

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-03-03 Thread Andres Freund
Hi, On 2022-03-03 13:11:17 -0500, Robert Haas wrote: > On Wed, Mar 2, 2022 at 3:00 PM Andres Freund wrote: > > On 2022-03-02 14:52:01 -0500, Robert Haas wrote: > > > - I am having some trouble understanding clearly what 0001 is doing. > > > I'll try to study it f

Re: casting operand to proper type in BlockIdGetBlockNumber

2022-03-03 Thread Andres Freund
ion). Which seems reasonable to combine with sanitizers? For CI I compared the cost of the different sanitizers. It looks like alignment sanitizer is almost free, undefined is pretty cheap, and address sanitizer is pretty expensive (but still much cheaper than valgrind). Greetings, Andres Fr

Re: casting operand to proper type in BlockIdGetBlockNumber

2022-03-03 Thread Andres Freund
Hi, On 2022-03-03 15:31:51 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-03-03 14:00:14 -0500, Tom Lane wrote: > > For the buildfarm, I could enable it on flaviventris? That runs an > > experimental gcc, without optimization (whereas serinus runs with > >

  1   2   3   4   5   6   7   8   9   10   >