Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Bharath Rupireddy
On Thu, Jan 26, 2023 at 12:35 AM Andres Freund wrote: > > Hi, > > On 2023-01-25 14:32:51 +0100, Jakub Wartak wrote: > > In other words it allows slow down of any backend activity. Any feedback on > > such a feature is welcome, including better GUC name proposals ;) and > > conditions in which such

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-26 Thread Masahiko Sawada
On Thu, Jan 26, 2023 at 3:54 PM John Naylor wrote: > > On Wed, Jan 25, 2023 at 8:42 AM Masahiko Sawada wrote: > > > > On Mon, Jan 23, 2023 at 8:20 PM John Naylor > > wrote: > > > > > > On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, Jan 16, 2023 at 2:02 PM

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-26 Thread Peter Eisentraut
On 24.01.23 07:57, Michael Paquier wrote: For the 0004 patch, it should be documented why one would want one behavior or the other. That's totally unclear right now. I am not 100% sure whether we should have this part at the end, but as an exit path in case somebody complains about the extra lo

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-26 Thread Peter Eisentraut
On 25.01.23 01:08, Michael Paquier wrote: On Tue, Jan 24, 2023 at 03:57:56PM +0900, Michael Paquier wrote: Makes sense. That would be my intention if 0004 is the most acceptable and splitting things makes things a bit easier to review. There was a silly mistake in 0004 where the jumbling code

Re: pg_upgrade from PG-14.5 to PG-15.1 failing due to non-existing function

2023-01-26 Thread David Geier
Hi, On 1/25/23 19:38, Dimos Stamatakis wrote: Hi hackers, I attempted to perform an upgrade from PG-14.5 to PG-15.1 with pg_upgrade and unfortunately it errors out because of a function that does not exist anymore in PG-15.1. The function is ‘pg_catalog.close_lb’ and it exists in 14.5 but

Re: improving user.c error messages

2023-01-26 Thread Alvaro Herrera
Please use errdetail("You must have %s privilege to create roles with %s.", "SUPERUSER", "SUPERUSER"))); in this kind of message where multiple copies appear that only differ in the keyword to use, to avoid creating four copies of essentially the same str

Re: meson oddities

2023-01-26 Thread Peter Eisentraut
On 19.01.23 21:45, Andres Freund wrote: Hi, On 2023-01-19 21:37:15 +0100, Peter Eisentraut wrote: On 11.01.23 12:05, Peter Eisentraut wrote: I think there is also an adjacent issue:  The subdir options may be absolute or relative.  So if you specify --prefix=/usr/local and --sysconfdir=/etc/po

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-26 Thread Drouvot, Bertrand
Hi, On 1/24/23 7:27 PM, Alvaro Herrera wrote: Looking again, I have two thoughts for making things easier: 1. I don't think wait_for_write_catchup is necessary, because calling wait_for_catchup() and omitting the 'mode' and 'lsn' arguments would already do the same thing. So what we should do

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-26 Thread Alvaro Herrera
On 2023-Jan-26, Drouvot, Bertrand wrote: > On 1/24/23 7:27 PM, Alvaro Herrera wrote: > > 1. I don't think wait_for_write_catchup is necessary, because > > calling wait_for_catchup() and omitting the 'mode' and 'lsn' arguments > > would already do the same thing. So what we should do is patch pla

Re: [PATCH] Make ON CONFLICT DO NOTHING and ON CONFLICT DO UPDATE consistent

2023-01-26 Thread Aleksander Alekseev
Hi Andres, > It *certainly* can't be right to just continue with the update in heap_update, I see no reason why. What makes this case so different from updating a tuple created by the previous command? > as you've done. You'd have to skip the update, not execute it. What am I > missing here? Si

Re: to_hex() for negative inputs

2023-01-26 Thread Dean Rasheed
On Wed, 25 Jan 2023 at 21:43, Peter Eisentraut wrote: > > On 24.01.23 14:10, Dean Rasheed wrote: > > I also think it might be useful for it to gain a couple of boolean options: > > > > 1). An option to output a signed value (defaulting to false, to > > preserve the current two's complement output)

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
On Tue, Aug 23, 2022 at 1:13 PM John Naylor wrote: > > On Tue, Aug 23, 2022 at 11:24 AM David Rowley wrote: > > > > On Tue, 23 Aug 2022 at 15:22, John Naylor wrote: > > > Did you happen to see > > > > > > https://www.postgresql.org/message-id/CAFBsxsFhq8VUSkUL5YO17cFXbCPwtbbxBu%2Bd9MFrrsssfDXm3Q

Re: [PATCH] Tracking statements entry timestamp in pg_stat_statements

2023-01-26 Thread Andrei Zubkov
Hi, The final version of this patch should fix meson build and tests. -- Andrei Zubkov From 94784bccd48a83cba58d6017253d0b8f051e159c Mon Sep 17 00:00:00 2001 From: Andrei Zubkov Date: Thu, 26 Jan 2023 13:18:11 +0300 Subject: [PATCH] pg_stat_statements: Track statement entry timestamp This patc

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Tomas Vondra
On 1/25/23 20:05, Andres Freund wrote: > Hi, > > On 2023-01-25 14:32:51 +0100, Jakub Wartak wrote: >> In other words it allows slow down of any backend activity. Any feedback on >> such a feature is welcome, including better GUC name proposals ;) and >> conditions in which such feature should b

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread Pavel Borisov
Hi, John! Generally, I like the separation of non-null values before sorting and would like to join as a reviewer when we come to patch. I have only a small question: > - Only if there is more than one sort key, qsort the null array. Ideally at > some point we would have a method of ignoring the

Re: drop postmaster symlink

2023-01-26 Thread Peter Eisentraut
On 26.01.23 01:03, Karl O. Pinc wrote: Buried in https://www.postgresql.org/message-id/20230107165942.748ccf4e%40slate.karlpinc.com is the one change I see that should be made. In doc/src/sgml/ref/allfiles.sgml at line 222 there is an ENTITY defined which references the deleted postmaster.sgml

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

2023-01-26 Thread David Geier
Hi, On 1/23/23 21:30, Andres Freund wrote: That's been the case since my first post in the thread :). Mainly, it seems easier to detect underflow cases during subtraction that way. And the factor of 2 in range doesn't change a whole lot. I just realized it the other day :). If you have time to

Re: Add LZ4 compression in pg_dump

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 11:24:47AM +, gkokola...@pm.me wrote: > I gave this a little bit of thought. I think that ReadHead should not > emit a warning, or at least not this warning as it is slightly misleading. > It implies that it will automatically turn off data restoration, which is > false.

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
On Thu, Jan 26, 2023 at 6:14 PM Pavel Borisov wrote: > > - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). > Should we need to sort by the s

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread David Rowley
On Thu, 26 Jan 2023 at 23:29, John Naylor wrote: > Coming back to this, I wanted to sketch out this idea in a bit more detail. > > Have two memtuple arrays, one for first sortkey null and one for first > sortkey non-null: > - Qsort the non-null array, including whatever specialization is availabl

wrong Append/MergeAppend elision?

2023-01-26 Thread Amit Langote
Hi, It seems that the planner currently elides an Append/MergeAppend that has run-time pruning info (part_prune_index) set, but which I think is a bug. Here's an example: create table p (a int) partition by list (a); create table p1 partition of p for values in (1); set plan_cache_mode to force_

Re: Record queryid when auto_explain.log_verbose is on

2023-01-26 Thread torikoshia
On 2023-01-26 12:40, Michael Paquier wrote: On Wed, Jan 25, 2023 at 04:46:36PM +0900, Michael Paquier wrote: Thanks. Will check and probably apply on HEAD. Done, after adding one test case with compute_query_id=regress and applying some indentation. -- Michael Thanks! On 2023-01-23 09:35,

Re: Inconsistency in vacuum behavior

2023-01-26 Thread Nikita Malakhov
Hi! Yes, I've checked that. What would be desirable behavior in the case above? Anyway, waiting for table unlock seems to be not quite right. On Sat, Jan 21, 2023 at 4:12 AM Nathan Bossart wrote: > On Mon, Jan 16, 2023 at 11:18:08AM +0300, Alexander Pyhalov wrote: > > Is it intended? Why don't

Re: pg_upgrade from PG-14.5 to PG-15.1 failing due to non-existing function

2023-01-26 Thread Dimos Stamatakis
## Dimos Stamatakis (dimos.stamata...@servicenow.com): > In our scenario we changed the permissions of this function in PG14.5 > (via an automated tool) and then pg_upgrade tries to change the > permissions in PG15.1 as well. Given that this function wasn't even documented and did nothing but thr

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Jakub Wartak
> On 1/25/23 20:05, Andres Freund wrote: > > Hi, > > > > Such a feature could be useful - but I don't think the current place of > > throttling has any hope of working reliably: [..] > > You're blocking in the middle of an XLOG insertion. [..] > Yeah, I agree the sleep would have to happen elsewher

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Wed, Jan 25, 2023 at 11:25 PM Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 7:41 PM Robert Haas wrote: > > Both Andres and I have repeatedly expressed concern about how much is > > being changed in the behavior of vacuum, and how quickly, and IMHO on > > the basis of very limited evidence t

Re: psql: Add role's membership options to the \du+ command

2023-01-26 Thread Pavel Luzanov
On 24.01.2023 20:16, David G. Johnston wrote: Yeah, I noticed the lack too, then went a bit too far afield with trying to compose a graph of the roles.  I'm still working on that but at this point it probably won't be something I try to get committed to psql.  Something more limited like this d

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Wed, Jan 25, 2023 at 10:56 PM Andres Freund wrote: > but that's only true because page level freezing neutered > vacuum_freeze_min_age. Compared to <16, it's a *huge* change. Do you think that page-level freezing (1de58df4fec7325d91f5a8345757314be7ac05da) was improvidently committed? I have a

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-26 Thread Melih Mutlu
Hi Shveta, Thanks for reviewing. shveta malik , 25 Oca 2023 Çar, 16:02 tarihinde şunu yazdı: > On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu > wrote: > --I see initial data copied, but new catalog columns srrelslotname > and srreloriginname are not updated: > postgres=# select sublastusedid from

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Andrew Dunstan
On 2023-01-23 Mo 09:49, Jelte Fennema wrote: > Indeed the flags you added are enough. Attached is a patch > that adds an updated pre-commit hook with the same behaviour > as the one before. I definitely think having a pre-commit hook > in the repo is beneficial, since writing one that works in all

Re: Non-superuser subscription owners

2023-01-26 Thread Robert Haas
On Wed, Jan 25, 2023 at 10:45 PM Jeff Davis wrote: > I propose that we have two predefined roles: pg_create_subscription, > and pg_create_connection. If creating a subscription with a connection > string, you'd need to be a member of both roles. But to create a > subscription with a server object,

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 12:08:16 +0100, Tomas Vondra wrote: > It's not clear to me how could it cause deadlocks, as we're not waiting > for a lock/resource locked by someone else, but it's certainly an issue > for uninterruptible hangs. Maybe not. But I wouldn't want to bet on it. It's a violation of a

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 14:40:56 +0100, Jakub Wartak wrote: > In summary: Attached is a slightly reworked version of this patch. > 1. Moved logic outside XLogInsertRecord() under ProcessInterrupts() > 2. Flushes up to the last page boundary, still uses SyncRepWaitForLSN() > 3. Removed GUC for now (alwa

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 13:33:27 +0530, Bharath Rupireddy wrote: > 6. Backends can ignore throttling for WAL records marked as unimportant, no? Why would that be a good idea? Not that it matters today, but those records still need to be flushed in case of a commit by another transaction. > 7. I think

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Jelte Fennema
On Thu, 26 Jan 2023 at 15:40, Andrew Dunstan wrote: > I didn't really like your hook, as it forces a reindent, and many people > won't want that (for reasons given elsewhere in this thread). I'm not sure what you mean by "forces a reindent". Like I explained you can simply run "git commit" again

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Wed, Jan 25, 2023 at 7:56 PM Andres Freund wrote: > > https://wiki.postgresql.org/wiki/Freezing/skipping_strategies_patch:_motivating_examples#Patch_2 > > > > The difference between this and VACUUM FREEZE is described here: > > > > "Note how we freeze most pages, but still leave a significant n

Re: [EXTERNAL] Re: Support load balancing in libpq

2023-01-26 Thread Jelte Fennema
After discussing this patch privately with Andres here's a new version of this patch. The major differences are: 1. Use the pointer value of the connection as a randomness source 2. Use more precise time as randomness source 3. Move addrinfo changes into a separate commit. This is both to make the

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 5:41 AM Robert Haas wrote: > On Wed, Jan 25, 2023 at 11:25 PM Peter Geoghegan wrote: > > On Wed, Jan 25, 2023 at 7:41 PM Robert Haas wrote: > > > Both Andres and I have repeatedly expressed concern about how much is > > > being changed in the behavior of vacuum, and how q

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 09:20:57 -0500, Robert Haas wrote: > On Wed, Jan 25, 2023 at 10:56 PM Andres Freund wrote: > > but that's only true because page level freezing neutered > > vacuum_freeze_min_age. Compared to <16, it's a *huge* change. > > Do you think that page-level freezing > (1de58df4fec732

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Andrew Dunstan
On 2023-01-26 Th 11:16, Jelte Fennema wrote: > On Thu, 26 Jan 2023 at 15:40, Andrew Dunstan wrote: >> I didn't really like your hook, as it forces a reindent, and many people >> won't want that (for reasons given elsewhere in this thread). > I'm not sure what you mean by "forces a reindent". Lik

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 8:35 AM Andres Freund wrote: > I think it's probably ok, but perhaps deserves a bit more thought about when > to "opportunistically" freeze. Perhaps to make it *more* aggressive than it's > now. > > With "opportunistic freezing" I mean freezing the page, even though we don'

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Jelte Fennema
> At this stage the files are now indented, so if it failed and you run > `git commit` again it will commit with the indention changes. No, because at no point a "git add" is happening, so the changes made by pgindent are not staged. As long as you don't run the second "git commit" with the -a fla

Re: Non-superuser subscription owners

2023-01-26 Thread Jeff Davis
On Thu, 2023-01-26 at 09:43 -0500, Robert Haas wrote: > I have no issue with that as a long-term plan. However, I think that > for right now we should just introduce pg_create_subscription. It > would make sense to add pg_create_connection in the same patch that > adds a CREATE CONNECTION command (

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 08:54:55 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 8:35 AM Andres Freund wrote: > > I think it's probably ok, but perhaps deserves a bit more thought about when > > to "opportunistically" freeze. Perhaps to make it *more* aggressive than > > it's > > now. > > > >

Re: fix and document CLUSTER privileges

2023-01-26 Thread Nathan Bossart
On Wed, Jan 25, 2023 at 08:27:57PM -0800, Jeff Davis wrote: > Committed these extra clarifications. Thank you. Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: meson oddities

2023-01-26 Thread Andres Freund
On 2023-01-26 10:20:58 +0100, Peter Eisentraut wrote: > On 19.01.23 21:45, Andres Freund wrote: > > Hi, > > > > On 2023-01-19 21:37:15 +0100, Peter Eisentraut wrote: > > > On 11.01.23 12:05, Peter Eisentraut wrote: > > > > I think there is also an adjacent issue:  The subdir options may be > > > >

Re: Add LZ4 compression in pg_dump

2023-01-26 Thread Justin Pryzby
On Wed, Jan 25, 2023 at 07:57:18PM +, gkokola...@pm.me wrote: > On Wednesday, January 25th, 2023 at 7:00 PM, Justin Pryzby > wrote: > > While looking at this, I realized that commit 5e73a6048 introduced a > > regression: > > > > @@ -3740,19 +3762,24 @@ ReadHead(ArchiveHandle *AH) > > > > -

Re: drop postmaster symlink

2023-01-26 Thread Karl O. Pinc
On Wed, 25 Jan 2023 18:03:25 -0600 "Karl O. Pinc" Buried in > https://www.postgresql.org/message-id/20230107165942.748ccf4e%40slate.karlpinc.com > is the one change I see that should be made. > > > In doc/src/sgml/ref/allfiles.sgml at line 222 there is an ENTITY > > defined which references the d

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > I assume the case you're thinking of is that pruning did *not* do any changes, > but in the process of figuring out that nothing needed to be pruned, we did a > MarkBufferDirtyHint(), and as part of that emitted an FPI? Yes. > > That's going

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Tomas Vondra
On 1/26/23 16:40, Andres Freund wrote: > Hi, > > On 2023-01-26 12:08:16 +0100, Tomas Vondra wrote: >> It's not clear to me how could it cause deadlocks, as we're not waiting >> for a lock/resource locked by someone else, but it's certainly an issue >> for uninterruptible hangs. > > Maybe not.

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
I wrote: >>> It'd probably be reasonable to file down that sharp edge by instead >>> specifying that TimestampDifferenceMilliseconds will clamp overflowing >>> differences to LONG_MAX. Maybe there should be a clamp on the underflow >>> side too ... but should it be to LONG_MIN or to zero? After l

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
Thanks for taking a look. On Thu, Jan 26, 2023 at 10:07:39AM +0100, Alvaro Herrera wrote: > Please use > errdetail("You must have %s privilege to create roles with %s.", > "SUPERUSER", "SUPERUSER"))); > > in this kind of message where multiple copies appear th

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Matthias van de Meent
On Thu, 26 Jan 2023 at 19:45, Peter Geoghegan wrote: > > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > I assume the case you're thinking of is that pruning did *not* do any > > changes, > > but in the process of figuring out that nothing needed to be pruned, we did > > a > > MarkBuff

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 11:35 AM Peter Geoghegan wrote: > You complained about the descriptions being theoretical. But there's > nothing theoretical about the fact that we more or less do *all* > freezing in an eventual aggressive VACUUM in many important cases, > including very simple cases like

Re: improving user.c error messages

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 2:14 PM Nathan Bossart wrote: > Yeah, it's probably better to say "to alter roles with %s" to refer to > roles that presently have the attribute and "to change the %s attribute" > when referring to privileges for the attribute. I did this in v2, too. > > I've also switched

Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
The symptom being exhibited by Michael's new BF animal tanager is perfectly reproducible elsewhere. $ cat /home/postgres/tmp/temp_config #default_toast_compression = lz4 wal_compression = lz4 $ export TEMP_CONFIG=/home/postgres/tmp/temp_config $ cd ~/pgsql/src/test/recovery $ make check PROVE_TEST

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-26 Thread Drouvot, Bertrand
Hi, On 1/26/23 10:42 AM, Alvaro Herrera wrote: On 2023-Jan-26, Drouvot, Bertrand wrote: On 1/24/23 7:27 PM, Alvaro Herrera wrote: 1. I don't think wait_for_write_catchup is necessary, because calling wait_for_catchup() and omitting the 'mode' and 'lsn' arguments would already do the same th

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 01:54:08PM -0500, Tom Lane wrote: > After looking closer, I see that TimestampDifferenceMilliseconds > already explicitly states that its output is intended for WaitLatch > and friends, which makes it perfectly sane for it to clamp the result > to [0, INT_MAX] rather than de

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 11:28 AM Robert Haas wrote: > I think it's pretty much impossible to freeze more aggressively > without losing in some scenario or other. If waiting longer to freeze > would have resulted in the data getting updated again or deleted > before we froze it, then waiting longer

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: > @@ -758,16 +776,13 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt) > { > /* things an unprivileged user certainly can't do */ > if (dinherit || dcreaterole || dcreatedb || dcanlogin || dconnlimit || > - dvalidUntil || disrep

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 01:54:08PM -0500, Tom Lane wrote: >> - * Both inputs must be ordinary finite timestamps (in current usage, >> - * they'll be results from GetCurrentTimestamp()). >> + * At least one input must be an ordinary finite timestamp, else the "diff" >> + *

Re: improving user.c error messages

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: >> Basically my question is whether having one error message for all of >> those cases is good enough, or whether we should be trying harder. I think the password case needs to be kept separate, because the cond

Re: Something is wrong with wal_compression

2023-01-26 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: > The symptom being exhibited by Michael's new BF animal tanager > is perfectly reproducible elsewhere. I think these tests have always failed with wal_compression ? https://www.postgresql.org/message-id/20210308.173242.463790587797836129.

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Justin Pryzby writes: > On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: >> The symptom being exhibited by Michael's new BF animal tanager >> is perfectly reproducible elsewhere. > I think these tests have always failed with wal_compression ? If that's a known problem, and we've done no

Re: wrong Append/MergeAppend elision?

2023-01-26 Thread David Rowley
On Fri, 27 Jan 2023 at 01:30, Amit Langote wrote: > It seems that the planner currently elides an Append/MergeAppend that > has run-time pruning info (part_prune_index) set, but which I think is > a bug. This is actually how I intended it to work. Whether it was a good idea or not, I'm currently

Re: Minimal logical decoding on standbys

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 18:56:10 +0100, Drouvot, Bertrand wrote: > - I'm struggling to create a test for btree killtuples as there is a need for > rows removal on the table (that could produce a conflict too): > Do you've a scenario in mind for this one? (and btw in what kind of WAL > record should th

Re: Something is wrong with wal_compression

2023-01-26 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 02:08:27PM -0600, Justin Pryzby wrote: > On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: > > The symptom being exhibited by Michael's new BF animal tanager > > is perfectly reproducible elsewhere. > > I think these tests have always failed with wal_compression ? >

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 03:04:30PM -0500, Tom Lane wrote: > Nathan Bossart writes: >> I wonder if we should explicitly reject negative timestamps to eliminate >> any chance of int64 overflow, too. > > Hmm. I'm disinclined to add an assumption that the epoch is in the past, > but I take your poin

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-26 Thread Reid Thompson
On Mon, 2023-01-23 at 12:31 -0800, Andres Freund wrote: > Hi, > > I think it's basically still waiting on author, until the O(N) cost is gone > from the overflow limit check. > > Greetings, > > Andres Freund Yes, just a rebase. There is still work to be done per earlier in the thread. I do wan

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 11:26 AM Matthias van de Meent wrote: > Could someone explain to me why we don't currently (optionally) > include the functionality of page freezing in the PRUNE records? I > think they're quite closely related (in that they both execute in > VACUUM and are required for lon

lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Tomas Vondra
Hi, I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) did not report any results for a couple days, and it seems it got into an infinite loop in REL_11_STABLE when building hash table in a parallel hashjoin, or something like that. It seems to be progressing now, probably b

Re: wrong Append/MergeAppend elision?

2023-01-26 Thread Tom Lane
David Rowley writes: > On Fri, 27 Jan 2023 at 01:30, Amit Langote wrote: >> It seems that the planner currently elides an Append/MergeAppend that >> has run-time pruning info (part_prune_index) set, but which I think is >> a bug. > There is still the trade-off of having to pull tuples through th

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 10:44:45 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > > That's going to be very significantly more aggressive. For example > > > it'll impact small tables very differently. > > > > Maybe it would be too aggressive, not sure. The cost o

Re: doc: add missing "id" attributes to extension packaging page

2023-01-26 Thread Brar Piening
On 18.01.2023 at 06:50, Brar Piening wrote: I'll give it a proper look this weekend. It turns out I didn't get a round tuit. ... and I'm afraid I probably will not be able to work on this until mid/end February so we'll have to move this to the next commitfest until somebody wants to take it

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Tom Lane
Tomas Vondra writes: > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > did not report any results for a couple days, and it seems it got into > an infinite loop in REL_11_STABLE when building hash table in a parallel > hashjoin, or something like that. > It seems to be

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 2:57 PM Peter Geoghegan wrote: > Relatively difficult for Andres, or for somebody else? What are the > real parameters here? Obviously there are no clear answers available. Andres is certainly smarter than the average guy, but practically any scenario that someone can crea

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 20:26:00 +0100, Matthias van de Meent wrote: > Could someone explain to me why we don't currently (optionally) > include the functionality of page freezing in the PRUNE records? I think we definitely should (and have argued for it a couple times). It's not just about reducing WA

Re: Non-superuser subscription owners

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 12:36 PM Jeff Davis wrote: > On Thu, 2023-01-26 at 09:43 -0500, Robert Haas wrote: > > I have no issue with that as a long-term plan. However, I think that > > for right now we should just introduce pg_create_subscription. It > > would make sense to add pg_create_connection

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 9:49 AM Tom Lane wrote: > Tomas Vondra writes: > > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > > did not report any results for a couple days, and it seems it got into > > an infinite loop in REL_11_STABLE when building hash table in a parall

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 12:54 PM Robert Haas wrote: > > The overwhelming cost is usually FPIs in any case. If you're not > > mostly focussing on that, you're focussing on the wrong thing. At > > least with larger tables. You just have to focus on the picture over > > time, across multiple VACUUM o

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 9:57 AM Thomas Munro wrote: > On Fri, Jan 27, 2023 at 9:49 AM Tom Lane wrote: > > Tomas Vondra writes: > > > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > > > did not report any results for a couple days, and it seems it got into > > > an infi

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 03:04:30PM -0500, Tom Lane wrote: >> Hmm. I'm disinclined to add an assumption that the epoch is in the past, >> but I take your point that the subtraction would overflow with >> TIMESTAMP_INFINITY and a negative finite timestamp. Maybe we should

Re: Cygwin cleanup

2023-01-26 Thread Justin Pryzby
Note that cirrus failed like this: https://api.cirrus-ci.com/v1/artifact/task/4881596411543552/testrun/build/testrun/subscription/010_truncate/log/010_truncate_publisher.log 2023-01-25 23:17:10.417 GMT [29821][walsender] [sub1][3/0:0] ERROR: could not open file "pg_logical/snapshots/0-14F2060.s

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 4:06 PM Peter Geoghegan wrote: > There is very good reason to believe that the large majority of all > data that people store in a system like Postgres is extremely cold > data: The systems where I end up troubleshooting problems seem to be, most typically, busy OLTP syste

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 04:09:51PM -0500, Tom Lane wrote: > Right, so more like this. LGTM -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Something is wrong with wal_compression

2023-01-26 Thread Andrey Borodin
On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: > > That test case is demonstrating fundamental > database corruption after a crash. > Not exactly corruption. XID was not persisted and buffer data did not hit a disk. Database is in the correct state. It was discussed long before WAL compression

Re: GUCs to control abbreviated sort keys

2023-01-26 Thread Peter Eisentraut
On 25.01.23 22:16, Jeff Davis wrote: I am highlighting this case because the existence of a single non- contrived case or regression suggests that we may want to explore further and tweak heuristics. That's quite natural when the heuristics are based on a complex dependency like a collation provi

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Andrew Dunstan
On 2023-01-26 Th 12:05, Jelte Fennema wrote: >> At this stage the files are now indented, so if it failed and you run >> `git commit` again it will commit with the indention changes. > No, because at no point a "git add" is happening, so the changes > made by pgindent are not staged. As long as y

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 1:22 PM Robert Haas wrote: > On Thu, Jan 26, 2023 at 4:06 PM Peter Geoghegan wrote: > > There is very good reason to believe that the large majority of all > > data that people store in a system like Postgres is extremely cold > > data: > > The systems where I end up troub

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 03:07:43PM -0500, Tom Lane wrote: > Nathan Bossart writes: >> On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: >>> Basically my question is whether having one error message for all of >>> those cases is good enough, or whether we should be trying harder. > > I

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Andrey Borodin writes: > On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: >> That test case is demonstrating fundamental >> database corruption after a crash. > Not exactly corruption. XID was not persisted and buffer data did not > hit a disk. Database is in the correct state. Really? I don't

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 04:09:51PM -0500, Tom Lane wrote: >> Right, so more like this. > LGTM Thanks, pushed. Returning to the prior patch ... I don't much care for this: +/* Maybe there will be a free slot in a second... */ +ret

Re: improving user.c error messages

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 03:07:43PM -0500, Tom Lane wrote: >> I think the password case needs to be kept separate, because the >> conditions for it are different (specifically the exception that >> you can alter your own password). Lumping the rest together >> seems OK to

Re: Something is wrong with wal_compression

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 11:14 AM Tom Lane wrote: > Andrey Borodin writes: > > On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: > >> That test case is demonstrating fundamental > >> database corruption after a crash. > > > Not exactly corruption. XID was not persisted and buffer data did not > >

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Jelte Fennema
On Thu, 26 Jan 2023 at 22:46, Andrew Dunstan wrote: > Hmm, but I usually run with -a, I even have a git alias for it. I guess > what this discussion illustrates is that there are various patters for > using git, and we shouldn't assume that everyone else is using the same > patterns we are. I def

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Thomas Munro writes: > On Fri, Jan 27, 2023 at 11:14 AM Tom Lane wrote: >> If any tuples made by that transaction had reached disk, >> we'd have a problem. > The problem is that the WAL wasn't flushed, allowing the same xid to > be allocated again after crash recovery. But for any data pages to

Re: GUCs to control abbreviated sort keys

2023-01-26 Thread Jeff Davis
On Thu, 2023-01-26 at 22:39 +0100, Peter Eisentraut wrote: > Maybe an easier way to enable or disable it in the source code with a > #define would serve this.  Making it a GUC right away seems a bit > heavy-handed.  Further exploration and tweaking might well require > further source code changes

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 12:45 PM Andres Freund wrote: > > Most of the overhead of FREEZE WAL records (with freeze plan > > deduplication and page-level freezing in) is generic WAL record header > > overhead. Your recent adversarial test case is going to choke on that, > > too. At least if you set

Re: Rework of collation code, extensibility

2023-01-26 Thread Jeff Davis
Attached v9 and added perf numbers below. I'm hoping to commit 0002 and 0003 soon-ish, maybe a week or two, please let me know if you want me to hold off. (I won't commit the GUCs unless others find them generally useful; they are included here to more easily reproduce my performance tests.) My

Partition key causes problem for volatile target list query

2023-01-26 Thread Bruce Momjian
I have found an odd behavior --- a query in the target list that assigns to a partitioned column causes queries that would normally be volatile to return always zero. In the first query, no partitioning is used: d1 | d2 + 1 | 0 2 | 0 2 | 1

  1   2   >