Re: Time to drop plpython2?

On Wed, Feb 16, 2022 at 11:14:46PM -0800, Andres Freund wrote: > I've pinged the owners of the animals failing so far: > - myna, butterflyfish These two are managed by a colleague, and I have an access to them. They should get back to green quickly now, if I did not mess up.. -- Michael signatur

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

On Thu, Feb 17, 2022 at 03:51:26PM +0900, Kyotaro Horiguchi wrote: > At Thu, 17 Feb 2022 14:58:38 +0900 (JST), Kyotaro Horiguchi > wrote in > > (Sorry for the broken mail...) > > > > > > > Ok, if -1 is wrong, what should the value of return if > > > somebody calling this function like: > > > pg_

Re: Make mesage at end-of-recovery less scary.

At Tue, 15 Feb 2022 20:17:20 +0530, Ashutosh Sharma wrote in > OK. The v13 patch looks good. I have marked it as ready to commit. > Thank you for working on all my review comments. Thaks! But the recent xlog.c refactoring crashes into this patch. And I found a silly bug while rebasing. xlog.c:

Re: Time to drop plpython2?

Hi, On 2022-02-16 22:52:08 -0800, Andres Freund wrote: > On 2022-02-16 11:58:50 -0800, Andres Freund wrote: > > Cool, will apply 1) later today. > > Done. Curious how red the BF will turn out to be. Let's hope it's not > too bad. I've pinged the owners of the animals failing so far: - snakefly,

Re: make tuplestore helper function

On Tue, Feb 15, 2022 at 11:57:56PM -0600, Justin Pryzby wrote: > On Tue, Jan 11, 2022 at 10:19:33AM -0500, Melanie Plageman wrote: >> I'll wait to do that if preferred by committer. >> Are you imagining that patch 0001 would only add the check for >> expectedDesc that is missing from pg_config() an

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

At Thu, 17 Feb 2022 15:51:26 +0900 (JST), Kyotaro Horiguchi wrote in > - Cosmic ray hit, or ill memory cell. 63 (0x3f) cannot be less than 42(0x2a) by one-bit flip. So the possibility of cosmic ray would be quite low. regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: O(n) tasks cause lengthy startups and checkpoints

Hi, On 2022-02-16 20:14:04 -0800, Nathan Bossart wrote: > >> - while ((spc_de = ReadDirExtended(spc_dir, "pg_tblspc", LOG)) != NULL) > >> + while (!ShutdownRequestPending && > >> + (spc_de = ReadDirExtended(spc_dir, "pg_tblspc", LOG)) != > >> NULL) > > > > Uh, huh? It strikes me as

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

> > The progress reporting mechanism of postgres uses the > > 'st_progress_param' array of 'PgBackendStatus' structure to hold the > > information related to the progress. There is a function > > 'pgstat_progress_update_param()' which takes 'index' and 'val' as > > arguments and updates the 'val' t

Re: Time to drop plpython2?

On 2022-02-16 11:58:50 -0800, Andres Freund wrote: > Cool, will apply 1) later today. Done. Curious how red the BF will turn out to be. Let's hope it's not too bad.

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

At Thu, 17 Feb 2022 14:58:38 +0900 (JST), Kyotaro Horiguchi wrote in > (Sorry for the broken mail...) > > At Wed, 16 Feb 2022 09:29:20 -0300, Ranier Vilela wrote > in > > > The patch: > > > pg_encoding_max_length(int encoding) > > > { > > > - Assert(PG_VALID_ENCODING(encoding)); > >

Re: adding 'zstd' as a compression algorithm

> 15 февр. 2022 г., в 23:20, Robert Haas написал(а): > > Anyway, those are my thoughts. What are yours? +1 for adding Zstd everywhere. Quite similar patches are already a part of "libpq compression" and "zstd for pg_dump" commitfest entries, so pushing this will simplify those entries IMV.

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

(Sorry for the broken mail...) At Wed, 16 Feb 2022 09:29:20 -0300, Ranier Vilela wrote in > > > ] > > > 633retval = pg_verify_mbstr_len(src_encoding, src_str, len, > > false); > > > 634 > > > > > > Trivial patch attached. > > > > Mmm? If the assert doesn't work, there should be inconcsi

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

At Wed, 16 Feb 2022 09:29:20 -0300, Ranier Vilela wrote in > > > ] > > > 633retval = pg_verify_mbstr_len(src_encoding, src_str, len, > > false); > > > 634 > > > > > > Trivial patch attached. > > > > Mmm? If the assert doesn't work, there should be inconcsistency > > between pg_enc and pg

Re: BufferAlloc: don't take two simultaneous locks

At Wed, 16 Feb 2022 10:40:56 +0300, Yura Sokolov wrote in > Hello, all. > > I thought about patch simplification, and tested version > without BufTable and dynahash api change at all. > > It performs suprisingly well. It is just a bit worse > than v1 since there is more contention around dynah

Re: Per-table storage parameters for TableAM/IndexAM extensions

On Sat, Feb 12, 2022 at 2:35 AM Robert Haas wrote: > > Imagine that I am using the "foo" tableam with "compression=lots" and > I want to switch to the "bar" AM which does not support that option. > If I remove the "compression=lots" option using a separate command, > the "foo" table AM may rewrit

Re: adding 'zstd' as a compression algorithm

On Wed, Feb 16, 2022 at 10:40:01AM -0500, Robert Haas wrote: > On Tue, Feb 15, 2022 at 9:33 PM Michael Paquier wrote: >> Yes, the patch misses the fact that each ./configure switch is >> documented. For consistency, I think that you should also add that in >> the MSVC scripts in the first version

Re: Race conditions in 019_replslot_limit.pl

Hi, On 2022-02-16 22:58:19 -0500, Tom Lane wrote: > I wonder though if an exiting walsender would always take that path. You're right - the CFI() in PostgresMain(), several replication commands, are not at all guaranteed to be in a transaction when throwing a FATAL. I don't quite see where we co

RE: [Proposal] Add foreign-server health checks infrastructure

> I understood here as removing following mechanism from core: > > * disable timeout at end of tx. While reading again and this part might be wrong. Sorry for inconvenience. But anyway some codes should be (re)moved from core, right? Best Regards, Hayato Kuroda FUJITSU LIMITED

Re: O(n) tasks cause lengthy startups and checkpoints

Hi Andres, I appreciate the feedback. On Wed, Feb 16, 2022 at 05:50:52PM -0800, Andres Freund wrote: >> +/* Since not using PG_TRY, must reset error stack by hand */ >> +if (sigsetjmp(local_sigjmp_buf, 1) != 0) >> +{ > > I also think it's a bad idea to introduce even more cop

RE: [Proposal] Add foreign-server health checks infrastructure

Dear Horiguchi-san, Thank you for giving your suggestions. I want to confirm your saying. > FWIW, I'm not sure this feature necessarily requires core support > dedicated to FDWs. The core have USER_TIMEOUT feature already and > FDWs are not necessarily connection based. It seems better if FDWs

Re: Race conditions in 019_replslot_limit.pl

Hi, On 2022-02-16 19:46:26 -0800, Andres Freund wrote: > But right now I'm not seeing what prevents us from throwing a FATAL error > while holding an lwlock? Nothing. But we register ShutdownPostgres() / ShutdownAuxiliaryProcess() to take care of that. I checked and walsenders do have ShutdownPo

Re: Race conditions in 019_replslot_limit.pl

Andres Freund writes: > But right now I'm not seeing what prevents us from throwing a FATAL error > while holding an lwlock? If we're inside a transaction, then ShutdownPostgres -> AbortOutOfAnyTransaction would release LWLocks. I wonder though if an exiting walsender would always take that path

Re: Race conditions in 019_replslot_limit.pl

Hi, On 2022-02-16 22:11:30 -0500, Tom Lane wrote: > So (a) it broke around 48 hours ago, which is already a useful > bit of info Indeed. 2f6501fa3c54bbe4568e3bcccd9a60d26a46b5ee seems like the obvious commit to blame. We document before_shmem_exit hooks as /* * Call before_shmem

Re: Nonrandom scanned_pages distorts pg_class.reltuples set by VACUUM

On Wed, Feb 16, 2022 at 7:08 PM Andres Freund wrote: > I'm quite worried about the pace and style of the vacuum changes. To me it > doesn't look like that there was design consensus before 44fa8488 was > committed, quite the opposite (while 44fa8488 probably isn't the center of > contention, it's

Re: Race conditions in 019_replslot_limit.pl

I wrote: > So (a) it broke around 48 hours ago, which is already a useful > bit of info, and (b) your animals seem far more susceptible than > anyone else's. Why do you suppose that is? Eyeballing the commit log for potential causes in that time frame, I can't help noticing 2 days ago Andre

Re: Race conditions in 019_replslot_limit.pl

Andres Freund writes: > On 2022-02-16 20:22:23 -0500, Tom Lane wrote: >> There's no disconnection log entry for either, which I suppose means >> that somebody didn't bother logging disconnection for walsenders ... > The thing is, we actually *do* log disconnection for walsenders: Ah, my mistake,

Re: Nonrandom scanned_pages distorts pg_class.reltuples set by VACUUM

Hi, On 2022-02-16 17:16:13 -0800, Peter Geoghegan wrote: > On Sun, Feb 13, 2022 at 1:43 PM Peter Geoghegan wrote: > > This fixes the observed problem directly. It also makes the code > > robust against other similar problems that might arise in the future. > > The risk that comes from trusting th

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 6:56 PM Robert Haas wrote: > I think that's not really what is happening, at least not in the cases > that typically are brought to my attention. In those cases, the > typical pattern is: > 5. None of the tables in the database have been vacuumed in a long > time. There ar

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 4:48 PM Peter Geoghegan wrote: > There might well be an element of survivorship bias here. Most VACUUM > operations won't ever attempt truncation (speaking very generally). > How many times might (say) the customer that John mentioned have > accidentally gone over xidStopLi

Re: Time to increase hash_mem_multiplier default?

On Tue, Feb 15, 2022 at 8:17 AM Justin Pryzby wrote: > The only reason not to is that a single-node hash-aggregate plan will now use > 2x work_mem. Which won't make sense to someone who doesn't deal with > complicated plans (and who doesn't know that work_mem is per-node and can be > used multipl

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022年2月17日(木) 10:56 Michael Paquier : > > On Wed, Feb 16, 2022 at 01:25:09PM +0900, Kasahara Tatsuhito wrote: > > Remove all references to tuplestore_donestoring() except for the header. > > Looks fine, thanks. This has no impact on Melanie's patch posted on > [1], so applied after tweaking the co

Re: killing perl2host

Hi, On 2022-02-16 14:42:30 -0800, Andres Freund wrote: > On February 16, 2022 1:10:35 PM PST, Andrew Dunstan > wrote: > >So something like this in Utils.pm: > > > > > >die "Msys targeted perl is unsupported for running TAP tests" if > >$Config{osname}eq 'msys'; > > I don't think we should rejec

Re: Race conditions in 019_replslot_limit.pl

Hi, On 2022-02-16 20:22:23 -0500, Tom Lane wrote: > There's no disconnection log entry for either, which I suppose means > that somebody didn't bother logging disconnection for walsenders ... The thing is, we actually *do* log disconnection for walsenders: 2022-02-16 17:53:55.780 PST [2459806][w

Re: libpq async duplicate error results

Andres Freund writes: > On 2022-02-16 20:28:02 -0500, Tom Lane wrote: >> Uh ... then we'd have to cast away the const to do free(). > I was just thinking of > if ((const PGresult *) res == &OOM_result) Oh, I see. Sure, can do it like that. regards, tom lane

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

On Wed, Feb 16, 2022 at 01:25:09PM +0900, Kasahara Tatsuhito wrote: > Remove all references to tuplestore_donestoring() except for the header. Looks fine, thanks. This has no impact on Melanie's patch posted on [1], so applied after tweaking the comment in tuplestore.h. [1]: https://www.postgre

Re: libpq async duplicate error results

Hi, On 2022-02-16 20:28:02 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-02-16 18:51:37 -0500, Tom Lane wrote: > >> + /* Also, do nothing if the argument is OOM_result */ > >> + if (res == unconstify(PGresult *, &OOM_result)) > >> + return; > > > Wouldn't it make more sens

Re: O(n) tasks cause lengthy startups and checkpoints

Hi, On 2022-02-16 16:50:57 -0800, Nathan Bossart wrote: > + * The custodian process is new as of Postgres 15. I think this kind of comment tends to age badly and not be very useful. > It's main purpose is to > + * offload tasks that could otherwise delay startup and checkpointing, but > + * it

Re: adding 'zstd' as a compression algorithm

On Tue, Feb 15, 2022 at 06:24:13PM -0800, Andres Freund wrote: > For backups it's pretty obviously zstd imo. At the lower levels it achieves > considerably higher compression ratios while still being vastly faster than > gzip. Without even using the threaded compression support the library has. No

Re: refactoring basebackup.c

Thanks for the comments Robert. I have addressed your comments in the attached patch v13-0002-ZSTD-add-server-side-compression-support.patch. Rest of the patches are similar to v12, but just bumped the version number. > It will be good if we can also fix > CreateWalTarMethod to support LZ4 and ZST

Re: Small TAP tests cleanup for Windows and unused modules

On Wed, Feb 16, 2022 at 05:36:14PM -0500, Andrew Dunstan wrote: > More specifically, all the tests in question are now passing on jacana > and fairywren, and their $Config{osname} is NOT 'msys' (it's 'MSWin32'). Indeed. 0001 is incorrect. > So we should simply remove any line that ends "if $Conf

Re: libpq async duplicate error results

Andres Freund writes: > On 2022-02-16 18:51:37 -0500, Tom Lane wrote: >> +/* Also, do nothing if the argument is OOM_result */ >> +if (res == unconstify(PGresult *, &OOM_result)) >> +return; > Wouldn't it make more sense to make res const, rather than unconstifying > &OOM_resu

Re: Race conditions in 019_replslot_limit.pl

Andres Freund writes: > I think the test is telling us that something may be broken. We shouldn't > silence that without at least some understanding what it is. I looked at the recent failure on komodoensis [1], and I think what is happening is just that the walsender for the basebackup run (laun

Re: Nonrandom scanned_pages distorts pg_class.reltuples set by VACUUM

On Sun, Feb 13, 2022 at 1:43 PM Peter Geoghegan wrote: > This fixes the observed problem directly. It also makes the code > robust against other similar problems that might arise in the future. > The risk that comes from trusting that scanned_pages is a truly random > sample (given these condition

Re: libpq async duplicate error results

Hi, On 2022-02-16 18:51:37 -0500, Tom Lane wrote: > This seems workable, and you'll notice it fixes the duplicated text > in the test case Andres was worried about. Cool. I find it mildly scary that we didn't have any other tests verifying the libpq side of connection termination. Seems like we

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

Here is a rebased patch. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 8fc744a0b22a1fff9f3470bfdc0e08d9dd5da5be Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 1 Dec 2021 23:50:49 + Subject: [PATCH v2 1/1] remove exclusive backup mode --- doc/src/sgml/backup.

Re: O(n) tasks cause lengthy startups and checkpoints

Here is another rebased patch set. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From c11a6893d2d509df1389a1c03081b6cc563d5683 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 5 Jan 2022 19:24:22 + Subject: [PATCH v5 1/8] Introduce custodian. The custodian process i

Re: pgsql: Add TAP test to automate the equivalent of check_guc

On Wed, Feb 16, 2022 at 07:39:26PM -0500, Robert Haas wrote: > Sorry, I saw that and then forgot about it ... but isn't it 3 days, > not 4 weeks? In any case, I'm happy to have you take care of it, but I > can also look at it tomorrow if you prefer. Ugh. I looked at the top of the thread and saw

Re: pgsql: Add TAP test to automate the equivalent of check_guc

On Wed, Feb 16, 2022 at 7:24 PM Michael Paquier wrote: > On Wed, Feb 16, 2022 at 01:38:51PM +0100, Christoph Berg wrote: > > The build works with the "Add TAP test to automate the equivalent of > > check_guc, take two" commit. Thanks! > > Thanks for double-checking, Christoph! > > > (Tests are sti

Re: pgsql: Add TAP test to automate the equivalent of check_guc

On Wed, Feb 16, 2022 at 01:38:51PM +0100, Christoph Berg wrote: > The build works with the "Add TAP test to automate the equivalent of > check_guc, take two" commit. Thanks! Thanks for double-checking, Christoph! > (Tests are still failing for > https://www.postgresql.org/message-id/YgjwrkEvNEqoz

Re: libpq async duplicate error results

I wrote: >>> An idea that I don't have time to pursue right now is to track >>> how much of conn->errorMessage has been read out by PQgetResult, >>> and only report newly-added text in later PQgetResult calls of >>> a series, while PQerrorMessage would continue to report the >>> whole thing. Buffe

Re: Small TAP tests cleanup for Windows and unused modules

Hi, On February 16, 2022 2:04:01 PM PST, Andrew Dunstan wrote: > >On 2/16/22 16:36, Daniel Gustafsson wrote: >> Seeing msys in TAP tests mentioned in a thread [1] tonight reminded me about >> two related (well, one of them) patches I had sitting around, so rather than >> forgetting again here ar

Re: killing perl2host

Hi, On February 16, 2022 1:10:35 PM PST, Andrew Dunstan wrote: > >On 2/16/22 16:01, Andres Freund wrote: >> Hi, >> >> On 2022-02-16 15:46:28 -0500, Andrew Dunstan wrote: >>> I suggest that we apply this patch: >> +1 >> >>> and if nothing breaks in a few days I will set about a more thorough >>>

Re: Small TAP tests cleanup for Windows and unused modules

On 2/16/22 17:04, Andrew Dunstan wrote: > On 2/16/22 16:36, Daniel Gustafsson wrote: >> Seeing msys in TAP tests mentioned in a thread [1] tonight reminded me about >> two related (well, one of them) patches I had sitting around, so rather than >> forgetting again here are some small cleanups. >>

Re: Small TAP tests cleanup for Windows and unused modules

> On 16 Feb 2022, at 23:04, Andrew Dunstan wrote: > > On 2/16/22 16:36, Daniel Gustafsson wrote: >> Seeing msys in TAP tests mentioned in a thread [1] tonight reminded me about >> two related (well, one of them) patches I had sitting around, so rather than >> forgetting again here are some small

Re: Small TAP tests cleanup for Windows and unused modules

On 2/16/22 16:36, Daniel Gustafsson wrote: > Seeing msys in TAP tests mentioned in a thread [1] tonight reminded me about > two related (well, one of them) patches I had sitting around, so rather than > forgetting again here are some small cleanups. > > 0001 attempts to streamline how we detect W

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 1:04 PM Robert Haas wrote: > No, what I'm saying is that people running older versions routinely > run VACUUM in single-user mode because otherwise it fails due to the > truncation issue. But once they go into single-user mode they lose > protection. Seems logically consis

Small TAP tests cleanup for Windows and unused modules

Seeing msys in TAP tests mentioned in a thread [1] tonight reminded me about two related (well, one of them) patches I had sitting around, so rather than forgetting again here are some small cleanups. 0001 attempts to streamline how we detect Windows in the TAP tests (after that there is a single

Re: adding 'zstd' as a compression algorithm (initdb/lz4)

On Tue, Feb 15, 2022 at 11:54:10AM -0800, Andres Freund wrote: > > Isn't it an incontrovertible fact that LZ4 is superior to pglz in > > every way? LZ4 is pretty much its successor. And so it seems totally > > fine to assume that users will always want to use the clearly better > > option, and that

Preventing indirection for IndexPageGetOpaque for known-size page special areas

Hi, I noticed that effectively all indexes use the special region of a page to store some index-specific data on the page. In all cases I've noticed, this is a constant-sized struct, located at what is effectively a fixed offset from the end of the page; indicated on the page by pd_special; and ac

Re: killing perl2host

On 2/16/22 16:01, Andres Freund wrote: > Hi, > > On 2022-02-16 15:46:28 -0500, Andrew Dunstan wrote: >> I suggest that we apply this patch: > +1 > >> and if nothing breaks in a few days I will set about a more thorough >> removal of perl2host() and adjusting everywhere it's called, and we can >>

Re: refactoring basebackup.c

On Wed, Feb 16, 2022 at 12:46 PM Jeevan Ladhe wrote: > So, I went ahead and have now also implemented client side decompression > for zstd. > > Robert separated[1] the ZSTD configure switch from my original patch > of server side compression and also added documentation related to > the switch. I

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 3:21 PM Peter Geoghegan wrote: > On Wed, Feb 16, 2022 at 12:11 PM Robert Haas wrote: > > No, I think it's PostgreSQL 13, because before the vacuum failsafe > > thing you could end up truncating enough tables during vacuum > > operations to actually wrap around. > > Why wou

Re: killing perl2host

Hi, On 2022-02-16 15:46:28 -0500, Andrew Dunstan wrote: > I suggest that we apply this patch: +1 > and if nothing breaks in a few days I will set about a more thorough > removal of perl2host() and adjusting everywhere it's called, and we can > forget that the whole sorry mess ever happened :-)

Re: USE_BARRIER_SMGRRELEASE on Linux?

On Wed, Feb 16, 2022 at 11:27:31AM -0800, Andres Freund wrote: > Did you check whether this is a problem recently introduced or long-lived? I've reproduced it back to v9.3. I'm assuming it's much older than that. > Does USE_BARRIER_SMGRRELEASE actually prevent this problem? Or was it just > that

killing perl2host

Largely following a recipe from Andres, I have migrated buildfarm animals fairywren and jacana to a setup that shouldn't need (and in fact won't be able to use) PostgreSQL::Test:Utils::perl2host(). AFAICT these two are the only buildfarm animals that run TAP tests under msys. See discussion at

Re: initdb / bootstrap design

Andres Freund writes: > On 2022-02-16 13:24:41 -0500, Tom Lane wrote: >> I remembered the reason why it's done that way: if we replaced those >> values during genbki.pl, the contents of postgres.bki would become >> architecture-dependent, belying its distribution as a "share" file. > Hm. Architec

Re: Time to drop plpython2?

On 2022-02-16 15:05:36 -0500, Chapman Flack wrote: > On 02/16/22 14:58, Andres Freund wrote: > >> "The minimum required version is Python 3.2 or > >> later." > > > > I stared for a bit, and I just don't see the redundancy? > > "minimum ... or later" maybe? Ah. Thanks.

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 12:11 PM Robert Haas wrote: > No, I think it's PostgreSQL 13, because before the vacuum failsafe > thing you could end up truncating enough tables during vacuum > operations to actually wrap around. Why wouldn't the xidStopLimit thing prevent actual incorrect answers to qu

Re: check-world has suddenly started spewing stuff on stderr

On 16/02/2022 19:04, Tom Lane wrote: $ make check-world >/dev/null 2022-02-16 11:57:47.548 EST [3339702] LOG: database system was not properly shut down; automatic recovery in progress 2022-02-16 11:57:47.612 EST [3339713] LOG: database system was not properly shut down; automatic recovery in

Re: Lowering the ever-growing heap->pd_lower

On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > wrote: > > Peter Geoghegan asked for good arguments for the two changes > > implemented. Below are my arguments detailed, with adversarial loads > > that show the problematic behaviou

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 2:18 PM Peter Geoghegan wrote: > It seems as if the advice about single user mode persisted for no > great reason at all. Technically there were some remaining reasons to > keep it around (like the truncation thing), but overall these > secondary reasons could have been add

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 1:28 PM Peter Geoghegan wrote: > On Wed, Feb 16, 2022 at 10:18 AM Andres Freund wrote: > > > I'm pretty sure that some people believe that wraparound can cause > > > actual data corruption > > > > Well, historically they're not wrong. > > True, but the most recent version

Re: Time to drop plpython2?

On 02/16/22 14:58, Andres Freund wrote: >> "The minimum required version is Python 3.2 or >> later." > > I stared for a bit, and I just don't see the redundancy? "minimum ... or later" maybe? Regards, -Chap

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

On Thu, 10 Feb 2022 at 07:53, Nitin Jadhav wrote: > > > > We need at least a trace of the number of buffers to sync (num_to_scan) > > > before the checkpoint start, instead of just emitting the stats at the > > > end. > > > > > > Bharat, it would be good to show the buffers synced counter and th

Re: Time to drop plpython2?

Hi, On 2022-02-16 15:43:19 +0100, Peter Eisentraut wrote: > On 15.02.22 22:40, Andres Freund wrote: > > 1: plpython: Reject Python 2 during build configuration. > > There is a bit of redundancy in the new wording in installation.sgml: > > "The minimum required version is Python 3.2 or > later."

Re: Lowering the ever-growing heap->pd_lower

On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent wrote: > Peter Geoghegan asked for good arguments for the two changes > implemented. Below are my arguments detailed, with adversarial loads > that show the problematic behaviour of the line pointer array that is > fixed with the patch. Why i

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

On Tue, 15 Feb 2022 at 13:16, Nitin Jadhav wrote: > > > Apart from above fields, I am planning to add few more fields to the > > view in the next patch. That is, process ID of the backend process > > which triggered a CHECKPOINT command, checkpoint start location, filed > > to indicate whether it

Re: initdb / bootstrap design

Hi, On 2022-02-16 13:24:41 -0500, Tom Lane wrote: > I remembered the reason why it's done that way: if we replaced those > values during genbki.pl, the contents of postgres.bki would become > architecture-dependent, belying its distribution as a "share" file. > While we don't absolutely have to co

Re: Observability in Postgres

Hi, On Wed, Feb 16, 2022 at 12:48:11AM -0500, Greg Stark wrote: > But on further thought I think what you're asking is whether there are > standard database metrics and standard names for them. A lot of this > work has already been done with pg_exporter but it is worth looking at > other database

Re: USE_BARRIER_SMGRRELEASE on Linux?

Hi, On 2022-02-16 09:37:21 -0800, Nathan Bossart wrote: > I've also figured out a > way to reproduce the issue without the pre-allocation patches applied: > > 1. In checkpointer.c, add a 30 second sleep before acquiring ckpt_lck to >increment ckpt_started. > 2. In session 1, run the following

Re: Observability in Postgres

On Wed, 2022-02-16 at 02:10 -0500, Greg Stark wrote: > On Tue, 15 Feb 2022 at 17:37, Magnus Hagander wrote: > > > But I think you'll run into a different problem much earlier. Pretty > > much everything out there is going to want to speak http(s). How are > > you going to terminate that, especial

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 10:27 AM Peter Geoghegan wrote: > True, but the most recent version where that's actually possible is > PostgreSQL 8.0, which was released in early 2005. It just occurred to me that the main historic reason for the single user mode advice was the lack of virtual XIDs. The

Re: Allow root ownership of client certificate key

Hi Tom, On 1/18/22 14:41, Tom Lane wrote: David Steele writes: [ client-key-perm-002.patch ] I took a quick look at this and agree with the proposed behavior change, but also with your self-criticisms: We may want to do the same on the server side to make the code blocks look more similar.

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 10:18 AM Andres Freund wrote: > > I'm pretty sure that some people believe that wraparound can cause > > actual data corruption > > Well, historically they're not wrong. True, but the most recent version where that's actually possible is PostgreSQL 8.0, which was released

Re: initdb / bootstrap design

Andres Freund writes: > Sure, there's a few tokens that we replace in initdb. As it turns out there's > only two rows that are actually variable. The username of the initial > superuser in pg_authid and the pg_database row for template 1, where encoding, > lc_collate and lc_ctype varies. The rest

Re: do only critical work during single-user vacuum?

Hi, On 2022-02-16 10:14:19 -0800, Peter Geoghegan wrote: > Absolutely -- couldn't agree more. Do you think it's worth targeting > 14 here, or just HEAD? I'd go for HEAD first, but wouldn't protest against 14. > I'm pretty sure that some people believe that wraparound can cause > actual data cor

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 9:56 AM Robert Haas wrote: > +1. But I think we might want to try to write documentation around > this. We should explicitly tell people NOT to use single-user mode, > because that stupid message has been there for a long time and a lot > of people have probably internalize

Re: PGEventProcs must not be allowed to break libpq

I wrote: > ... more generally, it seems to me that allowing a failing PGEventProc > to cause this to happen is just not sane. It breaks absolutely > every guarantee you might think we have about how libpq will behave. > As an example that seems very plausible currently, if an event proc > doesn't

improve --with-lz4 install documentation

Now that lz4 also supports backup compression, decompression, and more future enhancements that can be done here, I think it is better to make the --with-lz4 description more generic than adding specific details in there. Attached is the patch to remove the specifics related to WAL and TOAST compr

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 12:51 PM Peter Geoghegan wrote: > Good news! +1. But I think we might want to try to write documentation around this. We should explicitly tell people NOT to use single-user mode, because that stupid message has been there for a long time and a lot of people have probably

Re: initdb / bootstrap design

Hi, On 2022-02-16 11:47:31 +0100, Peter Eisentraut wrote: > On 16.02.22 03:12, Andres Freund wrote: > > Sure, there's a few tokens that we replace in initdb. As it turns out > > there's > > only two rows that are actually variable. The username of the initial > > superuser in pg_authid and the pg

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 8:48 AM Masahiko Sawada wrote: > FYI, I've tested the situation that I assumed autovacuum can not > correct the problem; when the system had already crossed xidStopLimit, > it keeps failing to vacuum on tables that appear in the front of the > list and have sufficient garba

Re: refactoring basebackup.c

Hi Everyone, So, I went ahead and have now also implemented client side decompression for zstd. Robert separated[1] the ZSTD configure switch from my original patch of server side compression and also added documentation related to the switch. I have included that patch here in the patch series f

Re: USE_BARRIER_SMGRRELEASE on Linux?

On Wed, Feb 16, 2022 at 08:44:42AM -0800, Nathan Bossart wrote: > On Tue, Feb 15, 2022 at 10:57:32PM -0800, Nathan Bossart wrote: >> On Tue, Feb 15, 2022 at 10:14:04PM -0800, Nathan Bossart wrote: >>> It looks like register_unlink_segment() is called prior to the checkpoint, >>> but the checkpointe

Re: Race conditions in 019_replslot_limit.pl

Hi, On 2022-02-16 18:04:14 +0900, Masahiko Sawada wrote: > On Wed, Feb 16, 2022 at 3:22 PM Kyotaro Horiguchi > wrote: > > > > At Wed, 16 Feb 2022 14:58:23 +0900, Masahiko Sawada > > wrote in > > > Or it's possible that the process took a time to clean up the > > > temporary replication slot? >

check-world has suddenly started spewing stuff on stderr

$ make check-world >/dev/null 2022-02-16 11:57:47.548 EST [3339702] LOG: database system was not properly shut down; automatic recovery in progress 2022-02-16 11:57:47.612 EST [3339713] LOG: database system was not properly shut down; automatic recovery in progress 2022-02-16 11:57:59.379 EST [

Re: Implementing Incremental View Maintenance

Hi, Nagata-san I am very interested in IMMV and read your patch but have some comments in v25-0007-Add-Incremental-View-Maintenance-support.patch and want to discuss with you. + /* For IMMV, we need to rewrite matview query */ + query = rewriteQueryForIMMV(query, into

Re: do only critical work during single-user vacuum?

On Wed, Feb 16, 2022 at 2:29 AM Peter Geoghegan wrote: > > On Mon, Feb 14, 2022 at 10:04 PM John Naylor > wrote: > > Well, the point of inventing this new vacuum mode was because I > > thought that upon reaching xidStopLimit, we couldn't issue commands, > > period, under the postmaster. If it was

Re: USE_BARRIER_SMGRRELEASE on Linux?

On Tue, Feb 15, 2022 at 10:57:32PM -0800, Nathan Bossart wrote: > On Tue, Feb 15, 2022 at 10:14:04PM -0800, Nathan Bossart wrote: >> It looks like register_unlink_segment() is called prior to the checkpoint, >> but the checkpointer is not calling RememberSyncRequest() until after >> SyncPreCheckpoi

Re: refactoring basebackup.c

On Wed, Feb 16, 2022 at 11:11 AM Alvaro Herrera wrote: > This is hard to interpret for humans though because of the nested > brackets and braces. It gets considerably easier if you split it in > separate variants: > >-Z, --compress=[{client|server}-]{gzip|lz4}[:LEVEL] >-Z, --compress=LEVE

  1   2   >