Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Wed, Nov 4, 2020 at 4:10 AM Magnus Hagander wrote: > On Wed, Oct 21, 2020 at 11:23 PM Thomas Munro wrote: > > Hmm, can we find a more descriptive name for this mechanism? What > > about calling it the "uncommitted enum table"? See attached. > > Thank

Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 1:12 PM Dagfinn Ilmari Mannsåker wrote: > Magnus Hagander writes: > > In looking at this I realize we also have exactly one thing referred to as > > "blacklist" in our codebase, which is the "enum blacklist" (and then a > > small internal variable in pgindent). > > Here's a

Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 1:44 PM Dagfinn Ilmari Mannsåker wrote: > Thanks! Just after sending that, I thought to grep for "white\W*list" > as well, and found a few more occurrences that were trivially reworded, > per the attached patch. Pushed.

Cirrus CI (Windows help wanted)

2021-01-04 Thread Thomas Munro
Hi, My new favourite CI is Cirrus CI, because it has 4 operating systems, generous enough quotas to handle 250+ branches in a single account, and public build/test log URLs. I flipped cfbot.cputube.org (mostly) over to that and it seems to work well so far -- fingers crossed. I've also been using

Re: Reducing WaitEventSet syscall churn

2021-01-04 Thread Thomas Munro
On Fri, Jul 31, 2020 at 9:42 AM Thomas Munro wrote: > On Thu, Jul 30, 2020 at 5:50 PM Thomas Munro wrote: > > I pushed those three patches, but will wait for more discussion on the rest. > > And here's a rebase, to make cfbot happy. And again. To restate the two goals of t

Re: Moving other hex functions to /common

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 4:47 PM Bruce Momjian wrote: > ... let's see how it likes this version. cfbot ideally processes a new patch fairly quickly but I didn't think of ".diff.gz" when writing the regexp to recognise patch files. I just tweaked the relevant regexp and it's building your patch now

Re: Context diffs

2021-01-05 Thread Thomas Munro
On Wed, Jan 6, 2021 at 6:58 AM Bruce Momjian wrote: > On Tue, Jan 5, 2021 at 11:21:07AM +1300, Thomas Munro wrote: > > For the record, cfbot just uses plain old GNU patch, because that > > seems to accept nearly everything that anyone posts here (after a step > > that tri

Re: Cirrus CI (Windows help wanted)

2021-01-05 Thread Thomas Munro
On Wed, Jan 6, 2021 at 11:48 AM Andrew Dunstan wrote: > OK, I have dug into this quite a bit. The way cirrus works is in fact > somewhat fragile. Anyway, here are the highlights: Thanks! > However, sadly the vctools package above isn't installed with all its > optional packages, so some crucial

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 12:55 PM Tom Lane wrote: > * Thomas' patch for improving timeout.c seems like a great idea, but > it did indeed have a race condition, and I felt the comments could do > with more work. Oops, and thanks! Very happy to see this one in the tree. > *

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 3:03 PM Thomas Munro wrote: > On Thu, Jan 7, 2021 at 12:55 PM Tom Lane wrote: > > * The SQLSTATE you chose for the new error condition seems pretty > > random. I do not see it in the SQL standard, so using a code that's > > within the spec-reserv

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 4:51 PM Tom Lane wrote: > Thomas Munro writes: > > One of the strange things about these errors is that they're > > asynchronous/unsolicited, but they appear to the client to be the > > response to their next request (if it doesn't eat ECONNR

Default wal_sync_method on FreeBSD

2021-01-07 Thread Thomas Munro
in our build farm that might qualify), so in that case we should just fall back to the regular decision logic that'll wind up using good old fsync(). From b5c7b0512ccbc1378803a8cc9a33fb9abf859eaf Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 8 Jan 2021 14:06:27 +1300 Subject: [PATC

Re: pgbench: option delaying queries till connections establishment?

2021-01-08 Thread Thomas Munro
On Sun, Jan 3, 2021 at 9:49 AM Fabien COELHO wrote: > > Just for fun, the attached 0002 patch is a quick prototype of a > > replacement for that stuff that seems to work OK on a Mac here. (I'm > > not sure if the Windows part makes sense or works.) > > Thanks! That will definitely help because I

Use pg_pwrite() in pg_test_fsync

2021-01-09 Thread Thomas Munro
Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 9 Jan 2021 23:37:18 +1300 Subject: [PATCH] Use pg_pwrite() in pg_test_fsync. For consistency with the PostgreSQL behavior we're tring to simulate, use pwrite() instead of lseek() + write(). --- src/bin/pg_test_fsync/pg_test_fsync.c | 43

Re: Key management with tests

2021-01-09 Thread Thomas Munro
On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian wrote: > Does anyone know why the cfbot applied the patch listed second first > here? > > http://cfbot.cputube.org/patch_31_2925.log > > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed > it would apply attachments in th

Re: new heapcheck contrib module

2021-01-10 Thread Thomas Munro
On Fri, Jan 8, 2021 at 6:33 AM Mark Dilger wrote: > The attached patches, v31, are mostly the same, but with "getopt_long.h" > included from pg_amcheck.c per Thomas's review, and a .gitignore file added > in contrib/pg_amcheck/ I couple more little things from Windows CI: C:\projects\postg

Re: [PATCH] Automatic HASH and LIST partition creation

2021-01-10 Thread Thomas Munro
On Wed, Oct 7, 2020 at 6:26 AM Anastasia Lubennikova wrote: > Do you think that it is a bug? For now, I removed this statement from > tests just to calm down the CI. I don't think we can use \d+ on a temporary table here, because the backend ID appears in the namespace, which is causing a failure

Experimenting with redo batching

2021-02-12 Thread Thomas Munro
le chain of them at once. From b481d7727d1732705b85e5935887ff4404eb014b Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 12 Feb 2021 01:08:04 +1300 Subject: [PATCH] Try to hold onto buffers between WAL records. When replaying a sequence of redo records that touch the same page, try to avoid having to look up, pin, loc

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 10:02 AM Peter Geoghegan wrote: > On Sat, Feb 6, 2021 at 7:40 PM Andres Freund wrote: > > Looks like a mistake on my part... Probably a rename regex that somehow > > went wrong - I went back and forth on those names way too many > > times. Want me to push the fix? > > Spot

Re: GlobalVisIsRemovableFullXid() vs GlobalVisCheckRemovableXid()

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 11:33 AM Peter Geoghegan wrote: > On Sun, Feb 14, 2021 at 2:08 PM Thomas Munro wrote: > > I prefer "next", because that's in the name of the variable it reads, > > and the variable name seemed to me to have a more obvious meaning. > >

GCC warning in back branches

2021-02-14 Thread Thomas Munro
); |^~~~ I propose the attached. From d0ce66b0a24c99ed94f8caac32084b44990863ce Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 15 Feb 2021 14:04:58 +1300 Subject: [PATCH 1/2] Fix compiler warning in back branches. Back-patch a tiny bit of commit fbb2e9a0 into 9.6 and 10, to silence an uninitialized

Re: doing something about the broken dynloader.h symlink

2021-02-14 Thread Thomas Munro
On Fri, Jun 19, 2020 at 11:38 PM Julien Rouhaud wrote: > On Fri, Jun 19, 2020 at 12:08 PM Thomas Munro wrote: > > On Fri, Jun 19, 2020 at 8:02 PM Peter Eisentraut > > wrote: > > > +[# Remove links created by old versions of configure, so that there > > > +# a

Re: shared tempfile was not removed on statement_timeout

2021-02-14 Thread Thomas Munro
On Fri, Feb 5, 2021 at 5:47 PM Thomas Munro wrote: > On Sun, Jan 31, 2021 at 6:07 PM Tom Lane wrote: > > Thomas Munro writes: > > > So that gives a very simple back-patchable patch. > > > > Hmm, so is the *rest* of that function perfectly okay with being > >

Re: Use pg_pwrite() in pg_test_fsync

2021-02-14 Thread Thomas Munro
On Sun, Jan 24, 2021 at 1:50 PM Thomas Munro wrote: > On Sun, Jan 10, 2021 at 9:21 AM Thomas Munro wrote: > > I left the fsync-after-closing and non-sync'd tests using write(), > > because they weren't using lseek(). The latter case is arguably a bit > > odd b

Re: Default wal_sync_method on FreeBSD

2021-02-14 Thread Thomas Munro
On Fri, Jan 8, 2021 at 2:19 PM Thomas Munro wrote: > The system header change has one interesting consequence for existing > releases of PostgreSQL, though: xlogdefs.h now sees that there is an > O_DSYNC macro that is distinct from O_SYNC, and defaults to > wal_sync_method=open_datas

Re: GCC warning in back branches

2021-02-14 Thread Thomas Munro
On Mon, Feb 15, 2021 at 2:35 PM Michael Paquier wrote: > ... I would have patched the top of the > function if it were me, btw. I just copied the way it is coded in master (due to commit fbb2e9a0 which fixed this warning in 11+).

Re: PG vs LLVM 12 on seawasp, next round

2021-02-14 Thread Thomas Munro
On Sat, Dec 12, 2020 at 8:45 AM Fabien COELHO wrote: > > +ERROR: could not load library > > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > > libLLVMOrcJIT.so.12git: cannot open shared object fi

Re: ERROR: invalid spinlock number: 0

2021-02-15 Thread Thomas Munro
On Mon, Feb 15, 2021 at 9:27 PM Michael Paquier wrote: > On Thu, Feb 11, 2021 at 11:30:13PM +0900, Fujii Masao wrote: > > Yes, so what about the attached patch? > > I see. So the first error triggering the spinlock error would cause > a transaction failure because the fallback implementation of a

Re: proposal - psql - use pager for \watch command

2021-02-15 Thread Thomas Munro
On Fri, Jan 8, 2021 at 10:36 PM Pavel Stehule wrote: > ne 19. 4. 2020 v 19:27 odesílatel Pavel Stehule > napsal: >> last week I finished pspg 3.0 https://github.com/okbob/pspg . pspg now >> supports pipes, named pipes very well. Today the pspg can be used as pager >> for output of \watch comma

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-16 Thread Thomas Munro
On Mon, Jan 18, 2021 at 11:22 AM Thomas Munro wrote: > On Mon, Jan 18, 2021 at 10:59 AM Justin Pryzby > |postgres=# SELECT > pg_collation_actual_version(123); > > |ERROR: cache lookup failed for collation 123 > > Yeah, not a great user experience. Will f

Re: Finding cause of test fails on the cfbot site

2021-02-17 Thread Thomas Munro
On Thu, Feb 18, 2021 at 9:18 AM Andrew Dunstan wrote: > On 2/17/21 11:06 AM, Tom Lane wrote: > > Peter Smith writes: > >> I saw that one of our commitfest entries (32/2914) is recently > >> reporting a fail on the cfbot site [1]. I thought this was all ok a > >> few days ago. > >> ... > >> Is the

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-17 Thread Thomas Munro
On Wed, Feb 17, 2021 at 8:04 PM Michael Paquier wrote: > On Wed, Feb 17, 2021 at 03:08:36PM +1300, Thomas Munro wrote: > > tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); > > if (!HeapTupleIsValid(tp)) > > + { > > +

Re: Finding cause of test fails on the cfbot site

2021-02-17 Thread Thomas Munro
On Thu, Feb 18, 2021 at 9:42 AM Thomas Munro wrote: > ... (there have been a few random terminations I can't explain > recently, but they look different, I think there was a Google Compute > Engine outage that might explain that), ... There's also occasionally a failure like

A qsort template

2021-02-17 Thread Thomas Munro
%2BhUKGKMQFVpjr106gRhwk6R-nXv0qOcTreZuQzxgpHESAL6dw%40mail.gmail.com From edfa27a45eca139d683bd0a02136e1da5a489243 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 17 Aug 2020 21:31:56 +1200 Subject: [PATCH 1/3] Add sort_template.h for making fast sort functions. Move our qsort implementation into a header t

Re: Finding cause of test fails on the cfbot site

2021-02-19 Thread Thomas Munro
On Sat, Feb 20, 2021 at 10:31 AM Peter Smith wrote: > Here is another related question about the cfbot error reporting - > > The main cfbot "status page" [1] still shows a couple of fails for the > 32/2914 (for freebsd & linux). But looking more closely, those fails > are not from the latest run.

Outdated description for effective_io_concurrency

2021-02-20 Thread Thomas Munro
Hello, Andrew Gierth pointed out that I left behind some outdated advice about RAID spindles in the GUC's extra description field, in commit b09ff536. Let's just drop that description. Patch attached. From bb2fad4c540b18ad1683f7d69783f9b2634b1da2 Mon Sep 17 00:00:00 2001 From: Th

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Thomas Munro
On Sat, Feb 20, 2021 at 11:33 AM Thomas Munro wrote: > On Sat, Feb 20, 2021 at 10:31 AM Peter Smith wrote: > > Here is another related question about the cfbot error reporting - > > > > The main cfbot "status page" [1] still shows a couple of fails for the > &g

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Thomas Munro
On Sat, Feb 20, 2021 at 3:54 AM Andrew Dunstan wrote: > here's a very small and simple (and possibly naive) POC patch that > demonstrates this and seems to do the right thing. As a small variation that might be more parallelism-friendly, would it be better to touch a file with a known name in an

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-21 Thread Thomas Munro
ion? Rght, fixed. From 1a3092e6ca386bac3292aa60c20e3b2a2ce08fff Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 17 Feb 2021 14:10:40 +1300 Subject: [PATCH v3 1/4] Hide internal error for pg_collation_actual_version(). Instead of an unsightly internal "cache lookup failed" messag

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-22 Thread Thomas Munro
On Mon, Feb 22, 2021 at 8:27 PM Michael Paquier wrote: > Looks good to me, thanks! Pushed, with one further small change: I realised that tab completion should use a "schema" query.

Re: Autovacuum worker doesn't immediately exit on postmaster death

2021-02-22 Thread Thomas Munro
On Fri, Dec 11, 2020 at 7:57 AM Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > The if-we're-going-to-delay-anyway path in vacuum_delay_point seems > > OK to add a touch more overhead to, though. > > Alright, for this part at least, seems like it'd be something like the > attache

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Thomas Munro
On Tue, Feb 23, 2021 at 11:03 PM Andres Freund wrote: > over the last ~year I spent a lot of time trying to figure out how we could > add AIO (asynchronous IO) and DIO (direct IO) support to postgres. While > there's still a *lot* of open questions, I think I now have a decent handle on > most of

Re: REINDEX backend filtering

2021-02-24 Thread Thomas Munro
On Thu, Feb 25, 2021 at 1:22 AM Julien Rouhaud wrote: > #define reindexHasFilter(x)((x & REINDEXOPT_COLL_NOT_CURRENT) != 0) It's better to use "(x) & ..." in macros to avoid weird operator precedence problems in future code. It seems like there are several different names for similar thi

Replace buffer I/O locks with condition variables (reviving an old patch)

2021-02-25 Thread Thomas Munro
C56Xng%40mail.gmail.com [2] https://www.postgresql.org/message-id/flat/20210223100344.llw5an2aklengrmn%40alap3.anarazel.de From 64af4b141b8b96cd18b4c9afa700880cd4d01d87 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 26 Feb 2021 18:01:28 +1300 Subject: [PATCH] Replace buffer I/O locks with

Re: Optimising latch signals

2021-02-26 Thread Thomas Munro
s soon if there are no objections. From acacbf06aa54b4b139553b08d7f8511b0aaa0331 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 8 Aug 2020 15:08:09 +1200 Subject: [PATCH v5 1/5] Optimize latches to send fewer signals. Don't send signals to processes that aren't currently sleepin

Remove latch.c workaround for Linux < 2.6.27

2021-02-26 Thread Thomas Munro
Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 27 Feb 2021 11:22:16 +1300 Subject: [PATCH] Remove latch.c workaround for Linux < 2.6.27. Ancient Linux had no EPOLL_CLOEXEC flag, so commit 82ebbeb0 added a separate code path with an fcntl() call. Kernels of that vintage are long gone. Now se

Re: Reducing WaitEventSet syscall churn

2021-02-26 Thread Thomas Munro
On Tue, Jan 5, 2021 at 6:10 PM Thomas Munro wrote: > For point 2, the question I am raising is: why should users get a > special FATAL message in some places and not others, for PM death? > However, if people are attached to that behaviour, we could still > achieve goal 1 witho

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-02-26 Thread Thomas Munro
ommitting those two. The 0003 patch to achieve $SUBJECT needs more discussion. From 51bb33c8755efa11d24595ed0a27fa0b387e8153 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 27 Feb 2021 15:40:11 +1300 Subject: [PATCH v4 1/3] Use condition variables for ProcSignalBarriers. Instead of a poll/sl

Re: Remove latch.c workaround for Linux < 2.6.27

2021-02-27 Thread Thomas Munro
On Sat, Feb 27, 2021 at 9:01 PM Heikki Linnakangas wrote: > On 27 February 2021 01:10:23 EET, Thomas Munro wrote: > >Commit 82ebbeb0 added a workaround for (already in 2017) ancient Linux > >kernels with no EPOLL_CLOEXEC. I don't see any such systems in the > >build f

Re: Remove latch.c workaround for Linux < 2.6.27

2021-02-28 Thread Thomas Munro
On Sat, Feb 27, 2021 at 9:30 PM Thomas Munro wrote: > On Sat, Feb 27, 2021 at 9:01 PM Heikki Linnakangas wrote: > > What happens if you try to try to compile or run on such an ancient kernel? > > Does it fall back to something else? Can you still make it work with > >

Re: Optimising latch signals

2021-02-28 Thread Thomas Munro
On Sat, Feb 27, 2021 at 12:04 AM Thomas Munro wrote: > I'm planning to commit this soon if there are no objections. Pushed, with the addition of an SFD_CLOEXEC flag for the signalfd. Time to watch the buildfarm to find out if my speculation about illumos is correct...

Re: Reducing WaitEventSet syscall churn

2021-02-28 Thread Thomas Munro
On Sat, Feb 27, 2021 at 2:48 PM Thomas Munro wrote: > Here's the alternative patch set, with no change to existing error > message behaviour. I'm going to commit this version and close this CF > item soon if there are no objections. Pushed. That leaves just walreceiver and p

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-02-28 Thread Thomas Munro
On Sat, Feb 27, 2021 at 4:14 PM Thomas Munro wrote: > Here's a new version. The condition variable patch 0001 fixes a bug: > CleanupProcSignalState() also needs to broadcast. The hunk that > allows the interrupt handlers to use CVs while you're already waiting > on a C

Re: Add client connection check during the execution of the query

2021-02-28 Thread Thomas Munro
things were mentioned earlier that still need to be tidied up. From 5f7c327c5896369b80529467a3f1f64eab690887 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 1 Mar 2021 18:08:23 +1300 Subject: [PATCH v3] Detect dropped connections while running queries. Provide a new optional GUC that can be us

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-01 Thread Thomas Munro
On Mon, Nov 16, 2020 at 8:56 PM Michael Paquier wrote: > On Thu, Sep 24, 2020 at 05:55:17PM +1200, Thomas Munro wrote: > > Right, RestoreArchivedFile() uses system(), so I guess it can hang > > around for a long time after unexpected postmaster exit on every OS if > > t

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-01 Thread Thomas Munro
ning 'false' can lead > to > -* very frequent retries, so try hard to make that an uncommon case. > -*/ > + smgrrelease(); > > Should this instead be in smgr.c to avoid setting a precedent for procsignal.c > to be littered with absorption functions

Why does the BF sometimes not dump regression.diffs?

2021-03-01 Thread Thomas Munro
Hello, I saw this failure after a recent commit (though the next build succeeded, and I don't yet have any particular reason to believe that the commits it blamed are at fault, we'll see...): https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gombessa&dt=2021-03-01%2004%3A58%3A17 Strangely,

Re: Why does the BF sometimes not dump regression.diffs?

2021-03-01 Thread Thomas Munro
On Tue, Mar 2, 2021 at 1:32 PM Andrew Dunstan wrote: > The version numbering is a bit misleading on fairywren, which, as it's a > machine I control runs from a git checkout, which clearly is later than > REL_11 even though that's what the version string says. Commit 13d2143 > fixed this. It was in

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-01 Thread Thomas Munro
On Tue, Mar 2, 2021 at 12:00 AM Thomas Munro wrote: > On Mon, Nov 16, 2020 at 8:56 PM Michael Paquier wrote: > > No objections with the two changes from pg_usleep() to WaitLatch() so > > they could be applied separately first. > > I thought about committing that first pa

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-01 Thread Thomas Munro
On Tue, Feb 2, 2021 at 11:16 AM Thomas Munro wrote: > Right, the checkpoint itself is probably worse than this > "close-all-your-files!" thing in some cases [...] I've been wondering what obscure hazards these "tombstone" (for want of a better word) files guard aga

Re: Parallel Full Hash Join

2021-03-02 Thread Thomas Munro
NER, scanning the hash table and emitting unmatched inner tuples. To align parallel and serial hash join, change ExecScanHashTableForUnmatched() to also scan HashMemoryChunks for the unmatched tuple scan instead of accessing tuples through the hash table buckets. Author: Melanie Plageman Author: Tho

Re: Optimising latch signals

2021-03-02 Thread Thomas Munro
On Mon, Mar 1, 2021 at 2:29 PM Thomas Munro wrote: > Time to watch the buildfarm to find out if my speculation about > illumos is correct... I just heard that damselfly's host has been decommissioned with no immediate plan for a replacement. That was the last of the Solaris-fam

Re: A qsort template

2021-03-02 Thread Thomas Munro
On Wed, Mar 3, 2021 at 10:25 AM Daniel Gustafsson wrote: > > On 18 Feb 2021, at 04:09, Thomas Munro wrote: > > In another thread[1], I proposed $SUBJECT, but then we found a better > > solution to that thread's specific problem. The general idea is still > > goo

Re: pgbench: option delaying queries till connections establishment?

2021-03-02 Thread Thomas Munro
On Sun, Jan 31, 2021 at 1:18 AM Fabien COELHO wrote: > I think it would be much more consistent to move all the thread complement > stuff there directly: Currently (v8) the windows implementation is in > pgbench and the MacOS implementation in port, which is quite messy. Hmm. Well this is totall

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-03 Thread Thomas Munro
On Thu, Mar 4, 2021 at 4:18 AM Daniel Gustafsson wrote: > > On 1 Mar 2021, at 12:54, Thomas Munro wrote: > Based on my (limited) experience with procsignalbarriers I think this patch is Help wanted: must have at least 14 years experience with ProcSignalBarrier! Yeah, I'm still fi

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-03 Thread Thomas Munro
On Tue, Mar 2, 2021 at 5:28 PM Thomas Munro wrote: > On Tue, Feb 2, 2021 at 11:16 AM Thomas Munro wrote: > > Right, the checkpoint itself is probably worse than this > > "close-all-your-files!" thing in some cases [...] > > I've been wondering what obscure

Re: pgbench: option delaying queries till connections establishment?

2021-03-04 Thread Thomas Munro
On Wed, Mar 3, 2021 at 6:23 PM Thomas Munro wrote: > On Sun, Jan 31, 2021 at 1:18 AM Fabien COELHO wrote: > > I think it would be much more consistent to move all the thread complement > > stuff there directly: Currently (v8) the windows implementation is in > >

Make relfile tombstone files conditional on WAL level

2021-03-04 Thread Thomas Munro
at/CA%2BhUKGLdemy2gBm80kz20GTe6hNVwoErE8KwcJk6-U56oStjtg%40mail.gmail.com From 61a15ed286a1fd824b4e2b4b689cbe6688930e6e Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 2 Mar 2021 16:09:51 +1300 Subject: [PATCH] Make relfile tombstone files conditional on WAL level. Traditionally we have left behind an empty

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-04 Thread Thomas Munro
On Thu, Mar 4, 2021 at 11:54 AM Thomas Munro wrote: > > I've been wondering what obscure hazards these "tombstone" (for want > > of a better word) files guard against, besides the one described in > > the comments for mdunlink(). I've been thinking

Re: Replace buffer I/O locks with condition variables (reviving an old patch)

2021-03-04 Thread Thomas Munro
On Fri, Feb 26, 2021 at 7:08 PM Thomas Munro wrote: > Back in 2016, Robert Haas proposed to replace I/O locks with condition > variables[1]. Condition variables went in and have found lots of > uses, but this patch to replace a bunch of LWLocks and some busy > looping did not. Si

Re: 011_crash_recovery.pl intermittently fails

2021-03-04 Thread Thomas Munro
On Fri, Mar 5, 2021 at 5:10 PM Tom Lane wrote: > I wrote: > > I'd be kind of inclined to remove this test script altogether, on the > > grounds that it's wasting cycles on a function that doesn't really > > do what is claimed (and we should remove the documentation claim, too). > > Alternatively,

Re: pgbench: option delaying queries till connections establishment?

2021-03-04 Thread Thomas Munro
On Thu, Mar 4, 2021 at 10:44 PM Thomas Munro wrote: > v10-0002-pgbench-Refactor-the-way-we-do-thread-portabilit.patch Here's a better version of that part. I don't yet know if it actually works on Windows... From 3aa63dfc086ab1f687ed668091a6bda8bf270fa7 Mon Sep 17 00:00:00 2001

Re: Which PG version does CVE-2021-20229 affected?

2021-03-05 Thread Thomas Kellerer
Michael Paquier schrieb am 05.03.2021 um 08:38: > On Fri, Mar 05, 2021 at 12:32:43AM -0700, bchen90 wrote: >> NVD link: >> >> https://nvd.nist.gov/vuln/detail/CVE-2021-20229#vulnCurrentDescriptionTitle > > This link includes incorrect information. CVE-2021-20229 is only a > problem in 13.0 and 13.

Re: 011_crash_recovery.pl intermittently fails

2021-03-05 Thread Thomas Munro
On Fri, Mar 5, 2021 at 5:40 PM Tom Lane wrote: > Thomas Munro writes: > > On Fri, Mar 5, 2021 at 5:10 PM Tom Lane wrote: > >> Alternatively, maybe we can salvage the function's usefulness by making it > >> flush WAL before returning? > > > To make pg_x

Re: Add client connection check during the execution of the query

2021-03-05 Thread Thomas Munro
On Mon, Mar 1, 2021 at 6:18 PM Thomas Munro wrote: > I've done a quick rebase of this the patch and added it to the > commitfest. No other changes. Several things were mentioned earlier > that still need to be tidied up. Rebased again due to bitrot. This time I did some actua

Re: is cfbot's apply aging intentional?

2021-03-07 Thread Thomas Munro
at is the intention at the moment? Is this a cfbot bug > > -- or just me being inadequately informed? ;) > > The cfbot will periodically try to rebuild all open patches on the current > (and > next) commitfest, as the main goal is to quickly spot patches that have > rotte

Re: 011_crash_recovery.pl intermittently fails

2021-03-07 Thread Thomas Munro
On Mon, Mar 8, 2021 at 1:39 PM Kyotaro Horiguchi wrote: > At Fri, 05 Mar 2021 11:16:55 -0500, Tom Lane wrote in > > Kyotaro Horiguchi writes: > > But, of course, first we need a fix for the bug we now know exists. > > Was anyone volunteering to make the patch? > &

Re: pgbench: option delaying queries till connections establishment?

2021-03-07 Thread Thomas Munro
On Fri, Mar 5, 2021 at 6:22 PM Thomas Munro wrote: > On Thu, Mar 4, 2021 at 10:44 PM Thomas Munro wrote: > > v10-0002-pgbench-Refactor-the-way-we-do-thread-portabilit.patch > > Here's a better version of that part. I don't yet know if it actually > works on Win

Re: Replace buffer I/O locks with condition variables (reviving an old patch)

2021-03-07 Thread Thomas Munro
On Fri, Mar 5, 2021 at 12:12 PM Thomas Munro wrote: > On Fri, Feb 26, 2021 at 7:08 PM Thomas Munro wrote: > > Back in 2016, Robert Haas proposed to replace I/O locks with condition > > variables[1]. Condition variables went in and have found lots of > > uses, but this pat

Re: Optimising latch signals

2021-03-08 Thread Thomas Munro
On Tue, Mar 9, 2021 at 12:20 PM Alvaro Herrera wrote: > On 2021-Mar-03, Thomas Munro wrote: > > On Mon, Mar 1, 2021 at 2:29 PM Thomas Munro wrote: > > > Time to watch the buildfarm to find out if my speculation about > > > illumos is correct... > > > > I ju

Re: Implementing Incremental View Maintenance

2021-03-08 Thread Thomas Munro
On Tue, Mar 9, 2021 at 1:22 PM Yugo NAGATA wrote: > On Mon, 8 Mar 2021 15:42:00 -0500 > Andrew Dunstan wrote: > > (A useful feature of the cfbot might be to notify the authors and > > reviewers when it detects bitrot for a previously passing entry.) > > +1 > The feature notifying it authors seems

Outdated comments about proc->sem in lwlock.c

2021-03-09 Thread Thomas Munro
Hi, In passing I noticed that lwlock.c contains 3 comments about bogus wakeups due to sharing proc->sem with the heavyweight lock manager and ProcWaitForSignal. Commit 675f55e (9.5) switched those things from proc->sem to proc->procLatch. ProcArrayGroupClearXid() and TransactionGroupUpdateXi

Re: Optimising latch signals

2021-03-09 Thread Thomas Munro
On Tue, Mar 9, 2021 at 1:09 PM Thomas Munro wrote: > On Tue, Mar 9, 2021 at 12:20 PM Alvaro Herrera > wrote: > > Hi, I don't know if you realized but we have two new Illumos members > > now (haddock and hake), and they're both failing initdb on signalfd() > > p

Re: pgbench: option delaying queries till connections establishment?

2021-03-09 Thread Thomas Munro
On Mon, Mar 8, 2021 at 3:18 PM Thomas Munro wrote: > David Rowley kindly tested this for me on Windows and told me how to > fix one of the macros that had incorrect error checking on that OS. > So here's a new version. I'm planning to commit 0001 and 0002 soon, > if t

Re: Replace buffer I/O locks with condition variables (reviving an old patch)

2021-03-10 Thread Thomas Munro
On Tue, Mar 9, 2021 at 6:24 PM Julien Rouhaud wrote: > > The old I/O lock array was the only user of struct > > LWLockMinimallyPadded, added in commit 6150a1b08a9, and it seems kinda > > strange to leave it in the tree with no user. Of course it's remotely > > possible there are extensions using

Re: fdatasync performance problem with large number of DB files

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 11:01 AM Michael Brown wrote: > * pg_basebackup receives a streaming backup (via [2] fsync_dir_recurse > or fsync_pgdata) unless --no-sync is specified > * postgres starts up unclean (via [3] SyncDataDirectory) > > We run multiple postgres clusters and some of those cluster

Re: fdatasync performance problem with large number of DB files

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 11:38 AM Thomas Munro wrote: > On Thu, Mar 11, 2021 at 11:01 AM Michael Brown > wrote: > > * is there a knob missing we can configure? > > * can we get an opt-in knob to use a single sync() call instead of a > > recursive fsync()? > > *

Re: fdatasync performance problem with large number of DB files

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 1:16 PM Tom Lane wrote: > Thomas Munro writes: > > Thinking about this some more, if you were to propose a patch like > > that syncfs() one but make it a configurable option, I'd personally be > > in favour of trying to squeeze it into v1

Re: fdatasync performance problem with large number of DB files

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 2:00 PM Fujii Masao wrote: > On 2021/03/11 8:30, Thomas Munro wrote: > > I've run into a couple of users who have just commented that recursive > > fsync() code out! > > BTW, we can skip that recursive fsync() by disabling fsync GUC even without

Re: fdatasync performance problem with large number of DB files

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 2:25 PM Tom Lane wrote: > Trolling the net, I found a newer-looking version of the man page, > and behold it says > >In mainline kernel versions prior to 5.8, syncfs() will fail only >when passed a bad file descriptor (EBADF). Since Linux 5.8, >sync

Re: Replace buffer I/O locks with condition variables (reviving an old patch)

2021-03-10 Thread Thomas Munro
On Thu, Mar 11, 2021 at 3:27 PM Julien Rouhaud wrote: > - /* > -* It would be nice to include the I/O locks in the BufferDesc, but that > -* would increase the size of a BufferDesc to more than one cache line, > -* and benchmarking has shown that keeping every BufferDesc aligned on a

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-10 Thread Thomas Munro
On Tue, Mar 2, 2021 at 2:10 PM Thomas Munro wrote: > ... One question I haven't > got to the bottom of: is it a problem for the startup process that CVs > use CHECK_FOR_INTERRUPTS()? This was a red herring. The startup process already reaches CFI() via various paths, as I figu

Re: 回复:Re: Cache relation sizes?

2021-03-11 Thread Thomas Munro
reSQL 14. So I will now move this to the next CF. Thanks for the reviews so far! From 735925fb72a8417e7940fca352877042266ae06b Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 13 Nov 2020 14:38:41 +1300 Subject: [PATCH v4 1/3] WIP: Track relation sizes in shared memory. MIME-Version: 1.0 Content-Type: text/plain; charset=U

A micro-optimisation for ProcSendSignal()

2021-03-11 Thread Thomas Munro
dSignal(). From b2f3bf47e11f572a32f24945d10ffb9a7bc47b1f Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 11 Mar 2021 23:09:11 +1300 Subject: [PATCH] Optimize ProcSendSignal(). Instead of referring to target processes by pid, use pgprocno so that we don't have to scan the ProcArray a

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Thomas Munro
While rebasing CF #2933 (which drops the _cached stuff and makes this optimisation always available, woo), I happened to notice that we're summing the size of many relations and forks into a variable nBlocksToInvalidate of type BlockNumber. That could overflow.

Re: compression libraries and CF bot

2021-03-11 Thread Thomas Munro
On Sun, Feb 21, 2021 at 4:30 AM Justin Pryzby wrote: > Dilip's TOAST patch is passing on linux and bsd --with-lz4, so I think it's > desirable to install on mac now. Justin figured this out, so now this patch is using lz4 and passing on Linux, FreeBSD and macOS. > libzstd would be desirable for

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread Thomas Munro
On Fri, Mar 12, 2021 at 5:20 PM Amit Kapila wrote: > uint64 +1

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-11 Thread Thomas Munro
On Thu, Mar 11, 2021 at 7:34 PM Michael Paquier wrote: > On Thu, Mar 11, 2021 at 04:37:39PM +1300, Thomas Munro wrote: > > Michael, when you said "That's pretty hack-ish, still efficient" in > > reference to this code: > > > >> - if (IsUnderPo

Re: pgbench - add pseudo-random permutation function

2021-03-12 Thread Thomas Munro
On Mon, Mar 8, 2021 at 11:50 PM Fabien COELHO wrote: > > I may have time to become familiar or at least semi-comfortable with all > > that weird math in it by then. > > Yep. > > Generating a parametric good-quality low-cost (but not > cryptographically-secure) pseudo-random permutations on arbitra

Re: Background writer and checkpointer in crash recovery

2021-03-12 Thread Thomas Munro
it for end-of-recovery checkpoint before allowing connections, then we'd better start creating tombstones in recovery unless the WAL level is high enough to avoid data eating hazards with unlogged changes and a double crash). [1] https://commitfest.postgresql.org/33/3030/ From 7a82c9e622ea45

<    1   2   3   4   5   6   7   8   9   10   >