RE: Logical replication timeout problem

2022-02-24 Thread kuroda.hay...@fujitsu.com
Dear Wang, Thank you for teaching some backgrounds about the patch. > According to our discussion, we need to send keepalive messages to subscriber > when skipping changes. > One approach is that **for each skipped change**, we try to send keepalive > message by calculating whether a timeout will

Re: [BUG] Panic due to incorrect missingContrecPtr after promotion

2022-02-24 Thread Kyotaro Horiguchi
At Thu, 24 Feb 2022 16:26:42 +0900 (JST), Kyotaro Horiguchi wrote in > So, actually WAL did not ended in an incomplete record. I think > FinishWalRecover is the last place to do that. (But it could be > earlier.) After some investigation, I finally concluded that we should reset abortedRecPtr

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Smith
Hi. Below are my review comments for the v2 patch. == 1. Commit message This patch changes the pg_stat_subscription_workers view (introduced by commit 8d74fc9) so that it stores only statistics counters: apply_error_count and sync_error_count, and has one entry for subscription. SUGGESTION

Re: Support logical replication of DDLs

2022-02-24 Thread Aleksander Alekseev
Hi Zheng, > >Also, I suspect that implementing it may be a bit challenging. What if we > >focus on table-level replication for now? > > I think it is due to the fact that the current limitations in logical > replication are > holding it back in major version upgrade (MVU). Online / reduced downtim

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Aleksander Alekseev
Hi Gunnar, > During a Patroni PR discussion > (https://github.com/zalando/patroni/pull/2225), we realised that if one > wants to use the "-c" option on a typical Debian/Ubuntu installation > (where the config resides below /etc/postgresql/), pg_rewind needs a way > to be told where the postgresql.

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread Aleksander Alekseev
Hi Samay, > I wanted to submit a patch to expose 2 new hooks (one for the authentication > check and another one for error reporting) in auth.c. These will allow users > to implement their own authentication methods for Postgres or add custom > logic around authentication. I like the idea - Po

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Eisentraut
On 21.02.22 17:17, Andres Freund wrote: Hi, On 2022-02-21 14:49:01 +0530, Amit Kapila wrote: On Mon, Feb 21, 2022 at 1:18 PM Andres Freund wrote: * stats_reset (Time at which these statistics were last reset) The view name could be pg_stat_subscription_lrep, pg_stat_logical_replication, or s

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Alexander Kukushkin
Hello Gunnar, On Thu, 24 Feb 2022 at 10:03, Aleksander Alekseev wrote: > > wants to use the "-c" option on a typical Debian/Ubuntu installation > > (where the config resides below /etc/postgresql/), pg_rewind needs a way > > to be told where the postgresql.conf actually is. > > > > The attached

Re: Typo in pgbench messages.

2022-02-24 Thread Tatsuo Ishii
>> I think you are right. In English there's should be no space between number >> and "%". >> AFAIK other parts of PostgreSQL follow the rule. I think it's better to back-patch this to stable branches if there's no objection. Thought? Best reagards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: h

Re: Extract epoch from Interval weird behavior

2022-02-24 Thread Aleksander Alekseev
Hi Joseph, > > Is this truncation on purpose? It seems like > > EXTRACT is not accounting for leap years in > > it's calculation. Extracting an epoch from an interval is quite a strange case since intervals are not connected to any specific dates. For instance: select extract('epoch' from inter

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Eisentraut
On 23.02.22 03:14, Andres Freund wrote: Why are the stats stored in the per-database stats file / as a second level below the database? While they're also associated with a database, it's a global catalog, so it seems to make more sense to have them "live" globally as well? pg_subscription bein

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Eisentraut
On 24.02.22 02:32, Masahiko Sawada wrote: On Wed, Feb 23, 2022 at 12:08 PM Peter Smith wrote: Hi. Below are my review comments for the v1 patch. Thank you for the comments! I've attached the latest version patch that incorporated all comments I got so far. The primary change from the previou

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Aleksander Alekseev
Hi hackers, > It took me a while to understand the meaning of -c. Maybe > changing it to --restore-target-wal will make it easier to > understand. +1, I "stumbled" while reading this too at first. -- Best regards, Aleksander Alekseev

Re: Readd use of TAP subtests

2022-02-24 Thread Peter Eisentraut
Now that we have switched everything to done_testing(), the subtests feature isn't that relevant anymore, but it might still be useful to get better output when running with PROVE_FLAGS=--verbose. Compare before: t/001_basic.pl .. 1..8 ok 1 - vacuumlo --help exit code 0 ok 2 - vacuumlo --help

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-24 Thread Amit Kapila
On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada wrote: > > Here are some comments: > > Why do we need SyncTableStartWrapper() and ApplyLoopWrapper()? > I have given this comment to move the related code to separate functions to slightly simplify ApplyWorkerMain() code but if you don't like we can

Re: make tuplestore helper function

2022-02-24 Thread Michael Paquier
On Mon, Feb 21, 2022 at 04:41:17PM +0900, Michael Paquier wrote: > So, I got my hands on this area, and found myself applying 07daca5 as > a first piece of the puzzle. Anyway, after more review today, I have > bumped into more pieces that could be consolidated, and finished with > the following pa

Re: Uniforms the errors msgs at tuplestore paths

2022-02-24 Thread Ranier Vilela
Em dom., 20 de fev. de 2022 às 22:45, Ranier Vilela escreveu: > Em dom., 20 de fev. de 2022 às 22:08, Michael Paquier > escreveu: > >> On Sun, Feb 20, 2022 at 11:37:33AM -0300, Ranier Vilela wrote: >> > I can't see: >> > plperl.c >> > pl_exec.c >> > pttcl.c >> > >> > Only jsonfuncs.c, but the er

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-24 Thread Michael Paquier
On Thu, Feb 24, 2022 at 12:15:40AM +, Jacob Champion wrote: > Stephen pointed out [1] that the authenticated identity that's stored > in MyProcPort can't be retrieved by extensions or triggers. Attached is > a patch that provides both a C API and a SQL function for retrieving > it. > > GetAuth

Re: Typo in pgbench messages.

2022-02-24 Thread Michael Paquier
On Thu, Feb 24, 2022 at 06:38:57PM +0900, Tatsuo Ishii wrote: > >> I think you are right. In English there's should be no space between > >> number and "%". > >> AFAIK other parts of PostgreSQL follow the rule. > > I think it's better to back-patch this to stable branches if there's > no objectio

Re: Uniforms the errors msgs at tuplestore paths

2022-02-24 Thread Michael Paquier
On Thu, Feb 24, 2022 at 08:30:02AM -0300, Ranier Vilela wrote: > Thanks for the commit Michael. No problem. For the archives, this is e77216f. -- Michael signature.asc Description: PGP signature

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 6:53 PM Peter Eisentraut wrote: > > On 24.02.22 02:32, Masahiko Sawada wrote: > > On Wed, Feb 23, 2022 at 12:08 PM Peter Smith wrote: > >> > >> Hi. Below are my review comments for the v1 patch. > > > > Thank you for the comments! I've attached the latest version patch > >

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Amit Kapila
On Thu, Feb 24, 2022 at 2:24 PM Peter Smith wrote: > > 9. src/backend/postmaster/pgstat.c - pgstat_recv_subscription_purge > > static void > pgstat_recv_subscription_purge(PgStat_MsgSubscriptionPurge *msg, int len) > { > /* Return if we don't have replication subscription statistics */ > if (subsc

Re: logical decoding and replication of sequences

2022-02-24 Thread Peter Eisentraut
On 23.02.22 12:10, Amit Kapila wrote: Isn't it better to support this with a syntax as indicated by Tom in one of his earlier emails on this topic [1]? IIUC, it would be as follows: CREATE PUBLICATION p FOR ALL TABLES, ALL SEQUENCES; I don't think there is any point in supporting this. What F

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 9:05 PM Amit Kapila wrote: > > On Thu, Feb 24, 2022 at 2:24 PM Peter Smith wrote: > > > > 10. src/backend/replication/logical/worker.c > > > > (from my previous [Peter-v1] #9) > > > > >> + /* Report the worker failed during table synchronization */ > > >> + pgstat_report_s

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Peter Eisentraut
On 18.12.21 00:53, Brar Piening wrote: The purpose is that you can directly link to the id in the public html docs which still gets generated (e. g. https://www.postgresql.org/docs/14/protocol-replication.html#PROTOCOL-REPLICATION-BASE-BACKUP). Essentially it gives people discussing the prot

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andrew Dunstan
On 2/23/22 20:52, Tom Lane wrote: > Peter Eisentraut writes: >> On 23.02.22 23:58, Tom Lane wrote: >>> Peter Eisentraut writes: libpq TAP tests should be in src/interfaces/libpq/t/. >>> That's failing to account for the fact that a libpq test can't >>> really be a pure-perl TAP test; you n

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Eisentraut
On 24.02.22 12:46, Masahiko Sawada wrote: We have a view called pg_stat_activity, which is very well known. From that perspective, "activity" means what is happening right now or what has happened most recently. The reworked view in this patch does not contain that (we already have pg_stat_su

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Peter Eisentraut
On 24.02.22 02:52, Tom Lane wrote: Peter Eisentraut writes: On 23.02.22 23:58, Tom Lane wrote: Peter Eisentraut writes: libpq TAP tests should be in src/interfaces/libpq/t/. That's failing to account for the fact that a libpq test can't really be a pure-perl TAP test; you need some C code

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread Andrew Dunstan
On 2/24/22 04:16, Aleksander Alekseev wrote: > Hi Samay, > >> I wanted to submit a patch to expose 2 new hooks (one for the authentication >> check and another one for error reporting) in auth.c. These will allow users >> to implement their own authentication methods for Postgres or add custom

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 9:23 PM Peter Eisentraut wrote: > > > On 24.02.22 12:46, Masahiko Sawada wrote: > >> We have a view called pg_stat_activity, which is very well known. From > >> that perspective, "activity" means what is happening right now or what > >> has happened most recently. The rew

Buffer Manager and Contention

2022-02-24 Thread Simon Riggs
Thinking about poor performance in the case where the data fits in RAM, but the working set is too big for shared_buffers, I notice a couple of things that seem bad in BufMgr, but don't understand why they are like that. 1. If we need to allocate a buffer to a new block we do this in one step, whi

Re: Typo in pgbench messages.

2022-02-24 Thread Fabien COELHO
Bonjour Michaël, I think it's better to back-patch this to stable branches if there's no objection. Thought? That's only cosmetic, so I would just bother about HEAD if I were to change something like that (I would not bother at all, personally). One argument against a backpatch is that this

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 8:08 PM Amit Kapila wrote: > > On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada wrote: > > > > Here are some comments: > > > > Why do we need SyncTableStartWrapper() and ApplyLoopWrapper()? > > > > I have given this comment to move the related code to separate > functions t

Re: Frontend error logging style

2022-02-24 Thread Peter Eisentraut
On 23.02.22 00:23, Tom Lane wrote: This conversation seems to have tailed off without full resolution, but I observe that pretty much everyone except Peter is on board with defining pg_log_fatal as pg_log_error + exit(1). So I think we should just do that, unless Peter wants to produce a finishe

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Daniel Gustafsson
> On 24 Feb 2022, at 10:27, Alexander Kukushkin wrote: > > Hello Gunnar, > > On Thu, 24 Feb 2022 at 10:03, Aleksander Alekseev > wrote: > > > wants to use the "-c" option on a typical Debian/Ubuntu installation > > (where the config resides below /etc/postgresql/), pg_rewind needs a way > >

Re: Readd use of TAP subtests

2022-02-24 Thread Daniel Gustafsson
> On 24 Feb 2022, at 11:16, Peter Eisentraut > wrote: > I think for deeply nested tests and test routines defined in other modules, > this helps structure the test output more like the test source code is laid > out, so it makes following the tests and locating failing test code easier. I don't

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Gunnar "Nick" Bluth
Am 24.02.2022 um 14:08 schrieb Daniel Gustafsson: On 24 Feb 2022, at 10:27, Alexander Kukushkin wrote: Hello Gunnar, On Thu, 24 Feb 2022 at 10:03, Aleksander Alekseev wrote: wants to use the "-c" option on a typical Debian/Ubuntu installation (where the config resides below /etc/postgres

Re: Typo in pgbench messages.

2022-02-24 Thread Daniel Gustafsson
> On 24 Feb 2022, at 13:58, Fabien COELHO wrote: >> One argument against a backpatch is that this would be disruptive with >> tools that parse and analyze the output generated by pgbench. Fabien, >> don't you have some tools and/or wrappers doing exactly that? > > Yep, I like to "| cut -d' ' -f

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-24 Thread Daniel Gustafsson
> On 24 Feb 2022, at 14:43, Gunnar Nick Bluth wrote: > That looks just as good to me, and it already has tests, so I'll happily step > down! Actually, I think this looks like a saner approach. Putting a config setting in two place (postgresql.conf and on the commandline for pg_rewind) is a rec

create_index test fails when synchronous_commit = off @ master

2022-02-24 Thread Aleksander Alekseev
Hi hackers, I noticed that create_index test (make installcheck) fails on my laptop because different query plans are chosen: - QUERY PLAN - Index Only Scan using tenk1_unique1 on tenk1 - Index Cond: (unique1 = ANY ('

Re: Patch a potential memory leak in describeOneTableDetails()

2022-02-24 Thread Daniel Gustafsson
> On 24 Feb 2022, at 07:32, Kyotaro Horiguchi wrote: > > At Thu, 24 Feb 2022 14:44:56 +0900 (JST), Kyotaro Horiguchi > wrote in >> At Tue, 22 Feb 2022 09:59:09 +0100, Daniel Gustafsson >> wrote in >>> The section in question was added to our docs 22 years ago, to make it >>> reflect >>> th

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Dagfinn Ilmari Mannsåker
Peter Eisentraut writes: > On 18.12.21 00:53, Brar Piening wrote: >> The purpose is that you can directly link to the id in the public html >> docs which still gets generated (e. g. >> https://www.postgresql.org/docs/14/protocol-replication.html#PROTOCOL-REPLICATION-BASE-BACKUP). >> >> Essentia

Re: GiST index build missing smgrimmedsync()?

2022-02-24 Thread Heikki Linnakangas
On 23/02/2022 23:30, Melanie Plageman wrote: I brought this up in [1] but wanted to start a dedicated thread. Since 16fa9b2b30a357 GiST indexes are not built in shared buffers. However, smgrimmedsync() is not done anywhere and skipFsync=true is always passed to smgrwrite() and smgrextend(). So,

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 13:23:55 +0100, Peter Eisentraut wrote: > On 24.02.22 12:46, Masahiko Sawada wrote: > > > We have a view called pg_stat_activity, which is very well known. From > > > that perspective, "activity" means what is happening right now or what > > > has happened most recently. The re

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 13:31:40 +0100, Peter Eisentraut wrote: > I think the TAP scripts should be in src/interfaces/libpq/t/, as usual. The > supporting code snippets could live in some other directory under > src/interfaces/libpq/, which might be called "test" or something else, not > that important.

Re: Frontend error logging style

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 14:06:18 +0100, Peter Eisentraut wrote: > My suggestion is to just get rid of pg_log_fatal() and replace them all with > pg_log_error(). -1. This ignores that already several places came up with their slightly different versions of fatal exit handlers. We don't gain anything by

Re: Readd use of TAP subtests

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 11:16:03 +0100, Peter Eisentraut wrote: > Now that we have switched everything to done_testing(), the subtests feature > isn't that relevant anymore, but it might still be useful to get better > output when running with PROVE_FLAGS=--verbose. I've incidentally played with subtes

Re: Frontend error logging style

2022-02-24 Thread Tom Lane
Peter Eisentraut writes: > This patch already illustrates a couple of things that are wrong with > this approach: All of these objections are a bit moot with my followup proposal, no? > - It doesn't allow any other way of exiting. For example, in pg_dump, > you have removed a few exit_nicely(

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Tom Lane
Andres Freund writes: > On 2022-02-24 13:31:40 +0100, Peter Eisentraut wrote: >> I think the TAP scripts should be in src/interfaces/libpq/t/, as usual. The >> supporting code snippets could live in some other directory under >> src/interfaces/libpq/, which might be called "test" or something else

Re: Extract epoch from Interval weird behavior

2022-02-24 Thread Joseph Koshakow
On Thu, Feb 24, 2022 at 4:47 AM Aleksander Alekseev wrote: > Extracting an epoch from an interval is quite a strange case since intervals > are not connected to any specific dates. I agree, I think it's a weird use case and that it's probably not worth fixing. Though it was fun for me to try. >

Re: create_index test fails when synchronous_commit = off @ master

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 16:47:25 +0300, Aleksander Alekseev wrote: > - QUERY PLAN > > - Index Only Scan using tenk1_thous_tenthous on tenk1 > - Index Cond: (thousand < 2) > - Filter: (tenthous = ANY ('{1001,3000}'::integer

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Alvaro Herrera
On 2022-Feb-24, Dagfinn Ilmari Mannsåker wrote: > Peter Eisentraut writes: > > Is there a way to obtain those URLs other than going into the HTML > > sources and checking if there is an anchor near where you want go? > > I use the jump-to-anchor extension: https://github.com/brettz9/jump-to-anc

Re: create_index test fails when synchronous_commit = off @ master

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 07:33:39 -0800, Andres Freund wrote: > I added the SELECT relpages, reltuples, relallvisible FROM pg_class WHERE oid > = 'tenk1'::regclass; > just after the > VACUUM ANALYZE tenk1; > > synchronous_commit=on > + relpages | reltuples | relallvisible > +--+---+-

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
Hi, Thank you for the comments! On Thu, Feb 24, 2022 at 4:20 PM tanghy.f...@fujitsu.com wrote: > > On Thu, Feb 24, 2022 9:33 AM Masahiko Sawada wrote: > > > > Thank you for the comments! I've attached the latest version patch > > that incorporated all comments I got so far. The primary change f

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Brar Piening
On 24.02.2022 at 16:46, Alvaro Herrera wrote: On 2022-Feb-24, Dagfinn Ilmari Mannsåker wrote: Peter Eisentraut writes: Is there a way to obtain those URLs other than going into the HTML sources and checking if there is an anchor near where you want go? I use the jump-to-anchor extension: htt

Re: Extract epoch from Interval weird behavior

2022-02-24 Thread Tom Lane
Joseph Koshakow writes: > I do want to briefly mention, if I'm understanding the history of > EXTRACT correctly, that the previous behavior > actually was to multiply by 365.25, not 365. However The commit that > changed the return type from numeric [1] > changed that behavior. Looking through the

Re: Time to drop plpython2?

2022-02-24 Thread Mark Wong
On Wed, Feb 23, 2022 at 07:59:01AM -0800, Mark Wong wrote: > On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote: > > Mark Wong writes: > > > Take 3. :) > > > > > I've upgraded everyone to the v14 buildfarm scripts and made sure the > > > --test passed on HEAD on each one. So I hopefully di

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 10:17:28 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-02-24 13:31:40 +0100, Peter Eisentraut wrote: > >> I think the TAP scripts should be in src/interfaces/libpq/t/, as usual. The > >> supporting code snippets could live in some other directory under > >> src/inte

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-24 Thread Jacob Champion
On Thu, 2022-02-24 at 20:39 +0900, Michael Paquier wrote: > I don't quite see the additional value that this API brings as > MyProcPort is directly accessible, and contrib modules like > postgres_fdw and sslinfo just use that to find the data of the current > backend. Right -- I just didn't know i

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Jacob Champion
On Thu, 2022-02-24 at 08:46 -0800, Andres Freund wrote: > One annoying bit is that our current tap invocation infrastructure for msvc > won't know how to deal with that. We put the build directory containing t/ > onto PATH, but that won't work for test/. But we also don't want to install > test bin

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-24 Thread Matthias van de Meent
On Wed, 23 Feb 2022 at 14:28, Nitin Jadhav wrote: > > Sharing the v2 patch. Kindly have a look and share your comments. Thanks for updating. > diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml With the new pg_stat_progress_checkpoint, you should also add a backreference t

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-24 Thread Julien Rouhaud
Hi, On Thu, Feb 24, 2022 at 04:50:59PM +, Jacob Champion wrote: > On Thu, 2022-02-24 at 20:39 +0900, Michael Paquier wrote: > > I don't quite see the additional value that this API brings as > > MyProcPort is directly accessible, and contrib modules like > > postgres_fdw and sslinfo just use t

Re: libpq async duplicate error results

2022-02-24 Thread Tom Lane
Actually ... it now seems to me that there's live bugs in the interaction between pipeline mode and error-buffer clearing. Namely, that places like PQsendQueryStart will unconditionally clear the buffer even though we may be in the middle of a pipeline sequence, and the buffer might hold an error t

Re: convert libpq uri-regress tests to tap test

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 17:03:33 +, Jacob Champion wrote: > On Thu, 2022-02-24 at 08:46 -0800, Andres Freund wrote: > > One annoying bit is that our current tap invocation infrastructure for msvc > > won't know how to deal with that. We put the build directory containing t/ > > onto PATH, but that w

Re: remove more archiving overhead

2022-02-24 Thread Nathan Bossart
Thanks for taking a look! On Thu, Feb 24, 2022 at 02:13:49PM +0900, Kyotaro Horiguchi wrote: > https://www.postgresql.org/docs/14/continuous-archiving.html > | The archive command should generally be designed to refuse to > | overwrite any pre-existing archive file. This is an important safety > |

Re: fix crash with Python 3.11

2022-02-24 Thread Tom Lane
I wrote: > * It might be a good idea to add parallel test cases for the other PLs. As I suspected, plperl and pltcl show exactly the same problems when trapping a failure at commit, reinforcing my opinion that this is a SPI bug that needs to be fixed in SPI. (plpgsql is not subject to this proble

Re: JSON path decimal literal syntax

2022-02-24 Thread Peter Eisentraut
On 18.02.22 11:17, Peter Eisentraut wrote: I noticed that the JSON path lexer does not support the decimal literal syntax forms .1 1. (that is, there are no digits before or after the decimal point).  This is allowed by the relevant ECMAScript standard (https://262.ecma-international.org/5.1

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-24 Thread Nitin Jadhav
> I think the change to ImmediateCheckpointRequested() makes no sense. > Before this patch, that function merely inquires whether there's an > immediate checkpoint queued. After this patch, it ... changes a > progress-reporting flag? I think it would make more sense to make the > progress-report

[PATCH] add relation and block-level filtering to pg_waldump

2022-02-24 Thread David Christensen
Greetings, This patch adds the ability to specify a RelFileNode and optional BlockNum to limit output of pg_waldump records to only those which match the given criteria. This should be more performant than `pg_waldump | grep` as well as more reliable given specific variations in output style d

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-24 Thread Peter Geoghegan
On Thu, Feb 24, 2022 at 11:06 AM David Christensen wrote: > This patch adds the ability to specify a RelFileNode and optional BlockNum to > limit output of > pg_waldump records to only those which match the given criteria. This should > be more performant > than `pg_waldump | grep` as well as m

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-24 Thread David Christensen
Added to commitfest as: https://commitfest.postgresql.org/37/3565/

Re: fix crash with Python 3.11

2022-02-24 Thread Tom Lane
I wrote: > * I'm not satisfied with using static storage for > SaveTransactionCharacteristics/RestoreTransactionCharacteristics. Looking closer at this, I was not too amused to discover that of the three existing SaveTransactionCharacteristics calls, two already conflict with each other: _SPI_comm

Re: document that brin's autosummarize parameter is off by default

2022-02-24 Thread Justin Pryzby
Ten months ago, Jaime Casanova wrote: > Hi everyone, > > Just noted that the default value of autosummarize reloption for brin > indexes is not documented, or at least not well documented. > > I added the default value in create_index.sgml where other options > mention their own defaults, also ma

Re: reallocing without oom check in pg_regress

2022-02-24 Thread Daniel Gustafsson
> On 23 Feb 2022, at 23:05, Tom Lane wrote: > > Daniel Gustafsson writes: >> In pg_regress we realloc() with the destination and source pointer being >> equal, >> without checking for OOM. While a fairly unlikely source of errors, is >> there a >> reason not to use pg_realloc() there for hygi

Re: Logical insert/update/delete WAL records for custom table AMs

2022-02-24 Thread Simon Riggs
On Mon, 17 Jan 2022 at 09:05, Jeff Davis wrote: > > On Wed, 2022-01-05 at 20:19 +, Simon Riggs wrote: > > I spoke with Jeff in detail about this patch in NYC Dec 21, and I now > > think it is worth pursuing. It seems much more likely that this would > > be acceptable than fully extensible rmgr

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-24 Thread Jacob Champion
On Fri, 2022-02-25 at 01:15 +0800, Julien Rouhaud wrote: > On Thu, Feb 24, 2022 at 04:50:59PM +, Jacob Champion wrote: > > On Thu, 2022-02-24 at 20:39 +0900, Michael Paquier wrote: > > > I don't quite see the additional value that this API brings as > > > MyProcPort is directly accessible, and

ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Tomas Vondra
Hi, Victor Yegorov reported a crash related to a GiST index on ltree [1], following a pg_upgrade from 12.x to 14.x, with a data set reproducing this. I spent some time investigating this, and it seems this is a silly bug in commit commit 911e70207703799605f5a0e8aad9f06cff067c63 (HEAD) Author:

real/float example for testlibpq3

2022-02-24 Thread Mark Wong
Hi everyone, Would adding additional examples to testlibpq3.c on handling more data types be helpful? I've attached a patch adding how to handle a REAL to current example. Regards, Mark diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c index 4f7b791388..0c1739fcbb 1006

Re: remove more archiving overhead

2022-02-24 Thread Nathan Bossart
I tested this again with many more WAL files and a much larger machine (r5d.24xlarge with data directory on an NVMe SSD instance store volume). As before, I am using a bare-bones archive module that does nothing but return true. Without the patch, archiving ~120k WAL files took about 109 seconds.

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Tom Lane
Tomas Vondra writes: > After thinking about this I only see two options: > 1) Don't apply the patch and tell everyone using ltree_gist they need to > rebuild the index after pg_upgrade from 12 to 13+. The downside of this > is larger indexes (because some tuples are 20B larger). > 2) Apply the p

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Andres Freund
Hi, On 2022-02-24 22:15:20 +0100, Tomas Vondra wrote: > After thinking about this I only see two options: > > 1) Don't apply the patch and tell everyone using ltree_gist they need to > rebuild the index after pg_upgrade from 12 to 13+. The downside of this > is larger indexes (because some tuples

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Tomas Vondra
On 2/24/22 23:06, Andres Freund wrote: > Hi, > > On 2022-02-24 22:15:20 +0100, Tomas Vondra wrote: >> After thinking about this I only see two options: >> >> 1) Don't apply the patch and tell everyone using ltree_gist they need to >> rebuild the index after pg_upgrade from 12 to 13+. The downsi

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Tomas Vondra
On 2/24/22 23:13, Tomas Vondra wrote: > > > On 2/24/22 23:06, Andres Freund wrote: >> Hi, >> >> On 2022-02-24 22:15:20 +0100, Tomas Vondra wrote: >>> After thinking about this I only see two options: >>> >>> 1) Don't apply the patch and tell everyone using ltree_gist they need to >>> rebuild t

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Thursday, February 24, 2022 11:07 AM Masahiko Sawada wrote: > I have some comments on v23 patch: > > @@ -66,6 +66,12 @@ typedef struct LogicalRepWorker > TimestampTz last_recv_time; > XLogRecPtr reply_lsn; > TimestampTz reply_time; > + > + /* > +* Transaction statistics of

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Wednesday, February 23, 2022 3:30 PM Amit Kapila > On Tue, Feb 22, 2022 at 6:45 AM tanghy.f...@fujitsu.com > wrote: > > > > I found a problem when using it. When a replication workers exits, the > > transaction stats should be sent to stats collector if they were not > > sent before because it

why do hash index builds use smgrextend() for new splitpoint pages

2022-02-24 Thread Melanie Plageman
I'm trying to understand why hash indexes are built primarily in shared buffers except when allocating a new splitpoint's worth of bucket pages -- which is done with smgrextend() directly in _hash_alloc_buckets(). Is this just so that the value returned by smgrnblocks() includes the new splitpoint

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread samay sharma
Hi Aleksander, On Thu, Feb 24, 2022 at 1:17 AM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Hi Samay, > > > I wanted to submit a patch to expose 2 new hooks (one for the > authentication check and another one for error reporting) in auth.c. These > will allow users to implement their

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Kyotaro Horiguchi
At Tue, 21 Dec 2021 08:47:27 +0100, Brar Piening wrote in > On 20.12.2021 at 16:09, Robert Haas wrote: > > As a data point, this is something I have also wanted to do, from time > > to time. I am generally of the opinion that any place the +1 from me. When I put an URL in the answer for inquiri

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Chapman Flack
On 02/24/22 19:52, Kyotaro Horiguchi wrote: > FWIW in that perspecive, there's no requirement from me that it should > be human-readable. I'm fine with automatically-generated ids. One thing I would be −many on, though, would be automatically-generated ids that are not, somehow, stable. I've been

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread Jeff Davis
On Thu, 2022-02-17 at 11:25 -0800, samay sharma wrote: > To enable this, I've proposed adding a new authentication method > "custom" which can be specified in pg_hba.conf and takes a mandatory > argument "provider" specifying which authentication provider to use. > I've also moved a couple static

Re: set TESTDIR from perl rather than Makefile

2022-02-24 Thread Justin Pryzby
On Mon, Feb 21, 2022 at 07:00:54AM -0500, Andrew Dunstan wrote: > On 2/19/22 18:53, Justin Pryzby wrote: > > On Sat, Feb 19, 2022 at 05:41:49PM -0600, Justin Pryzby wrote: > >> I rebased and fixed the check-guc script to work, made it work with vpath > >> builds, and cleaned it up some. > > I also

Re: Buffer Manager and Contention

2022-02-24 Thread Kyotaro Horiguchi
(I added Yura, as the author of a related patch) At Thu, 24 Feb 2022 12:58:23 +, Simon Riggs wrote in > Thinking about poor performance in the case where the data fits in > RAM, but the working set is too big for shared_buffers, I notice a > couple of things that seem bad in BufMgr, but don

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Tom Lane
Tomas Vondra writes: > I wonder if we could check the index tuple length, and adjust the siglen > based on that, somehow ... In an already-corrupted index, there won't be a unique answer. regards, tom lane

Re: Proposal: Support custom authentication methods using hooks

2022-02-24 Thread Tom Lane
Jeff Davis writes: > On Thu, 2022-02-17 at 11:25 -0800, samay sharma wrote: >> To enable this, I've proposed adding a new authentication method >> "custom" which can be specified in pg_hba.conf and takes a mandatory >> argument "provider" specifying which authentication provider to use. > One ca

Re: ltree_gist indexes broken after pg_upgrade from 12 to 13

2022-02-24 Thread Andres Freund
Hi, On February 24, 2022 5:44:57 PM PST, Tom Lane wrote: >Tomas Vondra writes: >> I wonder if we could check the index tuple length, and adjust the siglen >> based on that, somehow ... > >In an already-corrupted index, there won't be a unique answer. If we're breaking every ltree gist index, c

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Peter Smith
Below are my review comments for the v3 patch. == 1. Commit message (An earlier review comment [Peter-v2] #2 was only partly fixed) "there are no longer any relation information" --> "there is no longer any relation information" ~~~ 2. doc/src/sgml/monitoring.sgml + pg_stat_subscri

Re: Add id's to various elements in protocol.sgml

2022-02-24 Thread Maciek Sakrejda
On Thu, Feb 24, 2022, 16:52 Kyotaro Horiguchi wrote: > At Tue, 21 Dec 2021 08:47:27 +0100, Brar Piening wrote in > > On 20.12.2021 at 16:09, Robert Haas wrote: > > > As a data point, this is something I have also wanted to do, from time > > > to time. I am generally of the opinion that any place

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-24 Thread Amit Kapila
On Fri, Feb 25, 2022 at 4:41 AM Melanie Plageman wrote: > > I'm trying to understand why hash indexes are built primarily in shared > buffers except when allocating a new splitpoint's worth of bucket pages > -- which is done with smgrextend() directly in _hash_alloc_buckets(). > > Is this just so

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-24 Thread Amit Kapila
On Thu, Feb 24, 2022 at 12:03 PM Michael Paquier wrote: > > On Wed, Feb 23, 2022 at 09:52:02AM +0530, Amit Kapila wrote: > > Thanks, so you are okay with me pushing that patch just to HEAD. > > Yes, I am fine with that. I am wondering about patching the second > function though, to avoid any risk

Re: Buffer Manager and Contention

2022-02-24 Thread Kyotaro Horiguchi
At Fri, 25 Feb 2022 10:20:25 +0900 (JST), Kyotaro Horiguchi wrote in > (I added Yura, as the author of a related patch) > > At Thu, 24 Feb 2022 12:58:23 +, Simon Riggs > wrote in > > Thinking about poor performance in the case where the data fits in > > RAM, but the working set is too bi

  1   2   >