Re: speed up a logical replica setup

2024-07-30 Thread Euler Taveira
On Tue, Jul 30, 2024, at 7:35 AM, Peter Eisentraut wrote: > On 30.07.24 02:05, Euler Taveira wrote: > > I'm attaching a patch that implements option (c). While reading the code > > I noticed that I left a comment that should be removed by commit > > b9639138262. 0002 removes it. > > I have committ

Re: speed up a logical replica setup

2024-07-30 Thread Peter Eisentraut
On 30.07.24 02:05, Euler Taveira wrote: I'm attaching a patch that implements option (c). While reading the code I noticed that I left a comment that should be removed by commit b9639138262. 0002 removes it. I have committed your 0002 patch. It looks like Amit has already committed your 0001

RE: speed up a logical replica setup

2024-07-30 Thread Hayato Kuroda (Fujitsu)
Dear Euler, I applied your patch and confirmed it could fix the issue [1]. METHOD === 1. shortened the snapshot interval to 3ms, and ``` -#define LOG_SNAPSHOT_INTERVAL_MS 15000 +#define LOG_SNAPSHOT_INTERVAL_MS 3 ``` 2. ran test 040_pg_createsubscriber.pl many times. Before patching, I cou

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Amit Kapila
On Tue, Jul 30, 2024 at 11:28 AM Ashutosh Bapat wrote: > > On Tue, Jul 30, 2024 at 9:25 AM Amit Kapila wrote: > > > > On Tue, Jul 30, 2024 at 1:48 AM Tom Lane wrote: > > > > > > Robert Haas writes: > > > > On Sun, Jun 30, 2024 at 2:40 PM Tom Lane wrote: > > > >> ... However, I added a new open

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Ashutosh Bapat
On Tue, Jul 30, 2024 at 9:25 AM Amit Kapila wrote: > > On Tue, Jul 30, 2024 at 1:48 AM Tom Lane wrote: > > > > Robert Haas writes: > > > On Sun, Jun 30, 2024 at 2:40 PM Tom Lane wrote: > > >> ... However, I added a new open item about how the > > >> 040_pg_createsubscriber.pl test is slow and s

Re: speed up a logical replica setup

2024-07-29 Thread Amit Kapila
On Tue, Jul 30, 2024 at 9:26 AM Hayato Kuroda (Fujitsu) wrote: > > > Hayato already mentioned one of the solution in a previous email [2]. > > AFAICS we can use any solution that creates a WAL record. I tested the > > following options: > > Yes, my point was that we should add an arbitrary record

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Amit Kapila
On Tue, Jul 30, 2024 at 9:56 AM Tom Lane wrote: > > Amit Kapila writes: > > On Tue, Jul 30, 2024 at 1:48 AM Tom Lane wrote: > >> If the problem can be correctly described as "pg_createsubscriber > >> takes 10 seconds or so to detect end-of-stream", > > > The problem can be defined as: "pg_create

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Tom Lane
Amit Kapila writes: > On Tue, Jul 30, 2024 at 1:48 AM Tom Lane wrote: >> If the problem can be correctly described as "pg_createsubscriber >> takes 10 seconds or so to detect end-of-stream", > The problem can be defined as: "pg_createsubscriber waits for an > additional (new) WAL record to be ge

RE: speed up a logical replica setup

2024-07-29 Thread Hayato Kuroda (Fujitsu)
Dear Euler, > Hayato already mentioned one of the solution in a previous email [2]. > AFAICS we can use any solution that creates a WAL record. I tested the > following options: Yes, my point was that we should add an arbitrary record after the recovery_target_lsn. > (a) temporary replication s

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Amit Kapila
On Tue, Jul 30, 2024 at 1:48 AM Tom Lane wrote: > > Robert Haas writes: > > On Sun, Jun 30, 2024 at 2:40 PM Tom Lane wrote: > >> ... However, I added a new open item about how the > >> 040_pg_createsubscriber.pl test is slow and still unstable. > > > But that said, I see no commits in the commit

Re: speed up a logical replica setup

2024-07-29 Thread Euler Taveira
On Mon, Jul 29, 2024, at 6:11 PM, Euler Taveira wrote: > The options are: > > (a) temporary replication slot: requires an additional replication slot. > small payload. it is extremely slow in comparison with the other > options. > (b) logical message: can be consumed by logical replication when/if

Re: speed up a logical replica setup

2024-07-29 Thread Euler Taveira
On Wed, Jul 17, 2024, at 11:37 PM, Amit Kapila wrote: > I am thinking of transactions between restart_lsn and "consistent > point lsn" (aka the point after which all commits will be replicated). > You conclusion seems correct to me that such transactions won't be > replicated by streaming replicati

Re: 040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Tom Lane
Robert Haas writes: > On Sun, Jun 30, 2024 at 2:40 PM Tom Lane wrote: >> ... However, I added a new open item about how the >> 040_pg_createsubscriber.pl test is slow and still unstable. > But that said, I see no commits in the commit history which purport to > improve performance, so I guess th

040_pg_createsubscriber.pl is slow and unstable (was Re: speed up a logical replica setup)

2024-07-29 Thread Robert Haas
On Sun, Jun 30, 2024 at 2:40 PM Tom Lane wrote: > In hopes of moving things along as we approach the v18 branch, > I went ahead and pushed Kuroda-san's patches (with a bit of > further editorialization). AFAICS that allows closing out > the concerns raised by Noah, so I've marked that open item >

Re: speed up a logical replica setup

2024-07-17 Thread Amit Kapila
On Wed, Jul 17, 2024 at 5:28 PM Hayato Kuroda (Fujitsu) wrote: > > > Your analysis sounds correct to me. > > Okay, so we could have a same picture... > > > > IIUC, the root cause is that pg_create_logical_replication_slot() returns > > > a LSN > > > which is not generated yet. So, I think both mi

RE: speed up a logical replica setup

2024-07-17 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > Your analysis sounds correct to me. Okay, so we could have a same picture... > > IIUC, the root cause is that pg_create_logical_replication_slot() returns a > > LSN > > which is not generated yet. So, I think both mine [1] and Euler's approach > > [2] > > can solve the issue. My p

Re: speed up a logical replica setup

2024-07-17 Thread Amit Kapila
On Wed, Jul 17, 2024 at 1:23 PM Hayato Kuroda (Fujitsu) wrote: > > I also analyzed this failure, let me share it. Here, I think events in below > ordering were occurred. > > 1. Backend created a publication on $db2, > 2. BGWriter generated RUNNING_XACT record, then > 3. Backend created a replicati

RE: speed up a logical replica setup

2024-07-17 Thread Hayato Kuroda (Fujitsu)
Dear Alexander, Euler, Amit, I also analyzed this failure, let me share it. Here, I think events in below ordering were occurred. 1. Backend created a publication on $db2, 2. BGWriter generated RUNNING_XACT record, then 3. Backend created a replication slot on $db2. In this case, the recovery_ta

Re: speed up a logical replica setup

2024-07-14 Thread Amit Kapila
On Fri, Jul 12, 2024 at 4:54 AM Euler Taveira wrote: > > On Thu, Jul 11, 2024, at 2:00 PM, Alexander Lakhin wrote: > > May I ask you to look at another failure of the test occurred today [1]? > > > Thanks for the report! > > You are observing the same issue that Amit explained in [1]. The > pg_cre

Re: speed up a logical replica setup

2024-07-11 Thread Amit Kapila
On Thu, Jul 11, 2024 at 7:04 PM Euler Taveira wrote: > > On Thu, Jul 11, 2024, at 8:22 AM, Zhijie Hou (Fujitsu) wrote: > > The analysis and suggestion look reasonable to me. > Here is a small patch which does the same. > > > LGTM. However, I would add a comment above the INSERT you moved around to

Re: speed up a logical replica setup

2024-07-11 Thread Euler Taveira
On Thu, Jul 11, 2024, at 2:00 PM, Alexander Lakhin wrote: > Hello Amit and Hou-San, > > 11.07.2024 13:21, Amit Kapila wrote: > > > > We don't wait for the xmin to catch up corresponding to this insert > > and I don't know if there is a way to do that. So, we should move this > > Insert to after th

Re: speed up a logical replica setup

2024-07-11 Thread Alexander Lakhin
Hello Amit and Hou-San, 11.07.2024 13:21, Amit Kapila wrote: We don't wait for the xmin to catch up corresponding to this insert and I don't know if there is a way to do that. So, we should move this Insert to after the call to pg_sync_replication_slots(). It won't impact the general test of pg

Re: speed up a logical replica setup

2024-07-11 Thread Euler Taveira
On Thu, Jul 11, 2024, at 8:22 AM, Zhijie Hou (Fujitsu) wrote: > The analysis and suggestion look reasonable to me. > Here is a small patch which does the same. LGTM. However, I would add a comment above the INSERT you moved around to remind you that a transaction cannot be added there because it w

RE: speed up a logical replica setup

2024-07-11 Thread Zhijie Hou (Fujitsu)
On Thursday, July 11, 2024 6:21 PM Amit Kapila wrote: > > On Tue, Jul 9, 2024 at 4:30 PM Alexander Lakhin > wrote: > > > > Please look at another failure of the test [1]: > > [13:28:05.647](2.460s) not ok 26 - failover slot is synced > > [13:28:05.648](0.001s) # Failed test 'failover slot is s

Re: speed up a logical replica setup

2024-07-11 Thread Amit Kapila
On Wed, Jul 10, 2024 at 4:51 PM Euler Taveira wrote: > > On Tue, Jul 9, 2024, at 8:00 AM, Alexander Lakhin wrote: > > Hello Amit and Kuroda-san, > > 03.07.2024 14:02, Amit Kapila wrote: > > Pushed 0002 and 0003. Let's wait for a discussion on 0001. > > Please look at another failure of the test [1

Re: speed up a logical replica setup

2024-07-11 Thread Amit Kapila
On Tue, Jul 9, 2024 at 4:30 PM Alexander Lakhin wrote: > > Please look at another failure of the test [1]: > [13:28:05.647](2.460s) not ok 26 - failover slot is synced > [13:28:05.648](0.001s) # Failed test 'failover slot is synced' > # at > /home/bf/bf-build/skink-master/HEAD/pgsql/src/bin/p

Re: speed up a logical replica setup

2024-07-10 Thread Euler Taveira
On Tue, Jul 9, 2024, at 8:00 AM, Alexander Lakhin wrote: > Hello Amit and Kuroda-san, > > 03.07.2024 14:02, Amit Kapila wrote: > > Pushed 0002 and 0003. Let's wait for a discussion on 0001. > > Please look at another failure of the test [1]: > [13:28:05.647](2.460s) not ok 26 - failover slot is s

Re: speed up a logical replica setup

2024-07-09 Thread Alexander Lakhin
Hello Amit and Kuroda-san, 03.07.2024 14:02, Amit Kapila wrote: Pushed 0002 and 0003. Let's wait for a discussion on 0001. Please look at another failure of the test [1]: [13:28:05.647](2.460s) not ok 26 - failover slot is synced [13:28:05.648](0.001s) #   Failed test 'failover slot is synced'

Re: speed up a logical replica setup

2024-07-03 Thread Amit Kapila
On Wed, Jul 3, 2024 at 12:42 PM Hayato Kuroda (Fujitsu) wrote: > > > > > Disabling on the primary node should be sufficient. Let's do the > > minimum required to stabilize this test. > > +1, removed. > > PSA new version. 0001 has not been changed yet. A comment was added > in 0002 to clarify why w

RE: speed up a logical replica setup

2024-07-03 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > Your analysis looks correct to me. The test could fail due to > autovacuum. See the following comment in > 040_standby_failover_slots_sync. > > # Disable autovacuum to avoid generating xid during stats update as otherwise > # the new XID could then be replicated to standby at some ra

Re: speed up a logical replica setup

2024-07-02 Thread Amit Kapila
On Wed, Jul 3, 2024 at 10:42 AM Hayato Kuroda (Fujitsu) wrote: > > Based on that, I considered a scenario why the slot could not be synchronized. > I felt this was not caused by the pg_createsubscriber. > > 1. At initial stage, the xmin of the physical slot is 743, and nextXid of the >primary

Re: speed up a logical replica setup

2024-07-02 Thread Amit Kapila
On Wed, Jul 3, 2024 at 11:27 AM Amit Kapila wrote: > > > Do you have any other idea? > > > > The other idea could be that we use the minimum restart_lsn of all the > slots created by this tool as a consistent_lsn. We can probably get > that value by using pg_get_replication_slots() but this idea n

Re: speed up a logical replica setup

2024-07-02 Thread Amit Kapila
On Wed, Jul 3, 2024 at 9:21 AM Hayato Kuroda (Fujitsu) wrote: > > Dear Amit, > > > IIUC, the problem is that the consistent_lsn value returned by > > setup_publisher() is the "end +1" location of the required LSN whereas > > the recovery_target_lsn used in wait_for_end_recovery() expects the > > L

RE: speed up a logical replica setup

2024-07-02 Thread Hayato Kuroda (Fujitsu)
Dear Alexander and other hackers, > As a recent buildfarm failure [1] shows, that test addition introduced > new instability: > ### Starting node "node_s" > # Running: pg_ctl -w -D > /home/bf/bf-build/piculet/HEAD/pgsql.build/testrun/pg_basebackup/040_pg_c > reatesubscriber/data/t_040_pg_createsub

RE: speed up a logical replica setup

2024-07-02 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > IIUC, the problem is that the consistent_lsn value returned by > setup_publisher() is the "end +1" location of the required LSN whereas > the recovery_target_lsn used in wait_for_end_recovery() expects the > LSN value to be "start" location of required LSN. Yeah, right. It is same as

Re: speed up a logical replica setup

2024-07-02 Thread Amit Kapila
On Mon, Jul 1, 2024 at 8:22 PM Hayato Kuroda (Fujitsu) wrote: > > > I have a different but possibly-related complaint: why is > > 040_pg_createsubscriber.pl so miserably slow? On my machine it > > runs for a bit over 19 seconds, which seems completely out of line > > (for comparison, 010_pg_baseb

RE: speed up a logical replica setup

2024-07-01 Thread Hayato Kuroda (Fujitsu)
Dear Tom, > I have a different but possibly-related complaint: why is > 040_pg_createsubscriber.pl so miserably slow? On my machine it > runs for a bit over 19 seconds, which seems completely out of line > (for comparison, 010_pg_basebackup.pl takes 6 seconds, and the > other test scripts in this

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
Noah Misch writes: > On Sun, Jun 30, 2024 at 09:32:57PM -0400, Tom Lane wrote: >> Hmph ... according to [1], 545082091 was not enough to fix this. >> I guess that old version of IPC::Run also misbehaves for cases >> involving backslash, single quote, and/or some other special >> characters? > The

Re: speed up a logical replica setup

2024-06-30 Thread Noah Misch
On Sun, Jun 30, 2024 at 09:32:57PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Sun, Jun 30, 2024 at 05:01:52PM -0400, Tom Lane wrote: > >> I'm tempted to lobotomize the new test case on Windows until > >> we have that resolved. > > > Sounds fine. The pg_upgrade suite adequately tests appe

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
Noah Misch writes: > On Sun, Jun 30, 2024 at 05:01:52PM -0400, Tom Lane wrote: >> I'm tempted to lobotomize the new test case on Windows until >> we have that resolved. > Sounds fine. The pg_upgrade suite adequately tests appendShellString() and > appendConnStrVal() with the larger character rep

Re: speed up a logical replica setup

2024-06-30 Thread Noah Misch
On Sun, Jun 30, 2024 at 05:01:52PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Sun, Jun 30, 2024 at 02:58:00PM -0400, Tom Lane wrote: > >> So b3f5ccebd promptly blew up on fairywren [1]: > > > It does look consistent with IPC::Run predating v20220807.0, hence lacking > > the > > https://g

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
Noah Misch writes: > On Sun, Jun 30, 2024 at 02:58:00PM -0400, Tom Lane wrote: >> So b3f5ccebd promptly blew up on fairywren [1]: > It does look consistent with IPC::Run predating v20220807.0, hence lacking the > https://github.com/toddr/IPC-Run/issues/142 fix. I wondered how this animal > was p

Re: speed up a logical replica setup

2024-06-30 Thread Noah Misch
On Sun, Jun 30, 2024 at 02:58:00PM -0400, Tom Lane wrote: > I wrote: > > In hopes of moving things along as we approach the v18 branch, > > I went ahead and pushed Kuroda-san's patches (with a bit of > > further editorialization). > > So b3f5ccebd promptly blew up on fairywren [1]: > > connection

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
I wrote: > In hopes of moving things along as we approach the v18 branch, > I went ahead and pushed Kuroda-san's patches (with a bit of > further editorialization). So b3f5ccebd promptly blew up on fairywren [1]: connection error: 'psql: error: unterminated quoted string in connection info strin

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
In hopes of moving things along as we approach the v18 branch, I went ahead and pushed Kuroda-san's patches (with a bit of further editorialization). AFAICS that allows closing out the concerns raised by Noah, so I've marked that open item done. However, I added a new open item about how the 040_

Re: speed up a logical replica setup

2024-06-30 Thread Tom Lane
Alexander Lakhin writes: > As a recent buildfarm failure [1] shows, that test addition introduced > new instability: I have a different but possibly-related complaint: why is 040_pg_createsubscriber.pl so miserably slow? On my machine it runs for a bit over 19 seconds, which seems completely out

Re: speed up a logical replica setup

2024-06-30 Thread Alexander Lakhin
Hello Peter and Euler, 17.06.2024 14:04, Peter Eisentraut wrote: On 07.06.24 05:49, Euler Taveira wrote: Here it is a patch series to fix the issues reported in recent discussions. The patches 0001 and 0003 aim to fix the buildfarm issues. The patch 0002 removes synchronized failover slots on s

Re: speed up a logical replica setup

2024-06-27 Thread Amit Kapila
On Wed, Jun 26, 2024 at 11:35 AM Hayato Kuroda (Fujitsu) wrote: > > Thanks for giving comment! 0001 was modified per suggestions. > > > Yeah, it is a good idea to add a new option for two_phase but that > > should be done in the next version. For now, I suggest updating the > > docs and probably r

Re: speed up a logical replica setup

2024-06-26 Thread Noah Misch
On Tue, Jun 25, 2024 at 09:50:59PM -0300, Euler Taveira wrote: > On Tue, Jun 25, 2024, at 3:24 AM, Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 3:38 AM Noah Misch wrote: > > > On Mon, Jun 24, 2024 at 05:20:21PM +0530, Amit Kapila wrote: > > > > On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote:

RE: speed up a logical replica setup

2024-06-25 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Euler, Thanks for giving comment! 0001 was modified per suggestions. > Yeah, it is a good idea to add a new option for two_phase but that > should be done in the next version. For now, I suggest updating the > docs and probably raising a warning (if max_prepared_transactions != > 0) as

Re: speed up a logical replica setup

2024-06-25 Thread Amit Kapila
On Wed, Jun 26, 2024 at 7:21 AM Euler Taveira wrote: > > On Mon, Jun 24, 2024, at 3:47 AM, Hayato Kuroda (Fujitsu) wrote: > > > pg_createsubscriber fails on a dbname containing a space. Use > > appendConnStrVal() here and for other params in get_sub_conninfo(). See the > > CVE-2016-5424 commits

Re: speed up a logical replica setup

2024-06-25 Thread Euler Taveira
On Mon, Jun 24, 2024, at 3:47 AM, Hayato Kuroda (Fujitsu) wrote: > > pg_createsubscriber fails on a dbname containing a space. Use > > appendConnStrVal() here and for other params in get_sub_conninfo(). See the > > CVE-2016-5424 commits for more background. For one way to test this > > scenario,

Re: speed up a logical replica setup

2024-06-25 Thread Euler Taveira
On Tue, Jun 25, 2024, at 3:24 AM, Amit Kapila wrote: > On Tue, Jun 25, 2024 at 3:38 AM Noah Misch wrote: > > > > On Mon, Jun 24, 2024 at 05:20:21PM +0530, Amit Kapila wrote: > > > On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote: > > > > > > > > > +static void > > > > > +create_publication(PGcon

Re: speed up a logical replica setup

2024-06-24 Thread Amit Kapila
On Tue, Jun 25, 2024 at 3:38 AM Noah Misch wrote: > > On Mon, Jun 24, 2024 at 05:20:21PM +0530, Amit Kapila wrote: > > On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote: > > > > > > > +static void > > > > +create_publication(PGconn *conn, struct LogicalRepInfo *dbinfo) > > > > +{ > > > > > > > +

Re: speed up a logical replica setup

2024-06-24 Thread Noah Misch
On Mon, Jun 24, 2024 at 05:20:21PM +0530, Amit Kapila wrote: > On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote: > > > > > +static void > > > +create_publication(PGconn *conn, struct LogicalRepInfo *dbinfo) > > > +{ > > > > > + appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", >

Re: speed up a logical replica setup

2024-06-24 Thread Amit Kapila
On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote: > > > +static void > > +create_publication(PGconn *conn, struct LogicalRepInfo *dbinfo) > > +{ > > > + appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", > > + ipubname_esc); > > This tool's do

RE: speed up a logical replica setup

2024-06-23 Thread Hayato Kuroda (Fujitsu)
Dear Noah, > pg_createsubscriber fails on a dbname containing a space. Use > appendConnStrVal() here and for other params in get_sub_conninfo(). See the > CVE-2016-5424 commits for more background. For one way to test this > scenario, > see generate_db() in the pg_upgrade test suite. Thanks fo

Re: speed up a logical replica setup

2024-06-23 Thread Amit Kapila
On Sun, Jun 23, 2024 at 11:52 AM Noah Misch wrote: > > > +static char * > > +create_logical_replication_slot(PGconn *conn, struct LogicalRepInfo > > *dbinfo) > > +{ > > + PQExpBuffer str = createPQExpBuffer(); > > + PGresult *res = NULL; > > + const char *slot_name = dbinfo->replslo

Re: speed up a logical replica setup

2024-06-22 Thread Noah Misch
On Mon, Mar 25, 2024 at 12:55:39PM +0100, Peter Eisentraut wrote: > I have committed your version v33. > commit d44032d > --- /dev/null > +++ b/src/bin/pg_basebackup/pg_createsubscriber.c > +static char * > +concat_conninfo_dbname(const char *conninfo, const char *dbname) > +{ > + PQExpBuffe

Re: speed up a logical replica setup

2024-06-18 Thread Euler Taveira
On Mon, Jun 17, 2024, at 8:04 AM, Peter Eisentraut wrote: > On 07.06.24 05:49, Euler Taveira wrote: > > Here it is a patch series to fix the issues reported in recent > > discussions. The > > patches 0001 and 0003 aim to fix the buildfarm issues. The patch 0002 > > removes > > synchronized failov

Re: speed up a logical replica setup

2024-06-17 Thread Peter Eisentraut
On 07.06.24 11:17, Hayato Kuroda (Fujitsu) wrote: Other minor comments are included by the attached diff file. It contains changes to follow conventions and pgindent/pgperltidy. I have included some of your changes in the patches from Euler that I committed today. The 0004 patch might get rer

Re: speed up a logical replica setup

2024-06-17 Thread Peter Eisentraut
On 07.06.24 05:49, Euler Taveira wrote: Here it is a patch series to fix the issues reported in recent discussions. The patches 0001 and 0003 aim to fix the buildfarm issues. The patch 0002 removes synchronized failover slots on subscriber since it has no use. I also included an optional patch

RE: speed up a logical replica setup

2024-06-07 Thread Hayato Kuroda (Fujitsu)
Dear Euler, Thanks for making the follow-up patch! I was looking forward to your updates. I think this patch set is the solution for the found buildfarm error. However, there are remained claims raised by others. You should reply what you think for them. At least: 1) There are some misleading mes

Re: speed up a logical replica setup

2024-06-06 Thread Euler Taveira
On Wed, May 22, 2024, at 12:16 PM, Euler Taveira wrote: > I'll summarize all issues as soon as I finish the review of sync slot > support. I > think we should avoid new development if we judge that the item can be > documented as a limitation for this version. Nevertheless, I will share > patches

Re: speed up a logical replica setup

2024-05-24 Thread Shlok Kyal
On Wed, 22 May 2024 at 16:50, Amit Kapila wrote: > > On Mon, May 20, 2024 at 4:30 PM Shlok Kyal wrote: > > > > > > I was trying to test this utility when 'sync_replication_slots' is on > > > and it gets in an ERROR loop [1] and never finishes. Please find the > > > postgresql.auto used on the sta

Re: speed up a logical replica setup

2024-05-23 Thread Amit Kapila
On Thu, May 23, 2024 at 8:43 PM Euler Taveira wrote: > > On Thu, May 23, 2024, at 5:54 AM, Amit Kapila wrote: > > > Why in the first place do we need to ensure that primary_slot_name is > active on the primary? You mentioned something related to WAL > retention but I don't know how that is related

Re: speed up a logical replica setup

2024-05-23 Thread Euler Taveira
On Thu, May 23, 2024, at 5:54 AM, Amit Kapila wrote: > On Wed, May 22, 2024 at 8:46 PM Euler Taveira wrote: > > > > Following the same line that simplifies the code, we can: (a) add a loop in > > check_subscriber() that waits until walreceiver is available on subscriber > > or > > (b) use a timeo

Re: speed up a logical replica setup

2024-05-23 Thread Amit Kapila
On Wed, May 22, 2024 at 8:46 PM Euler Taveira wrote: > > On Wed, May 22, 2024, at 8:19 AM, Amit Kapila wrote: > > > v2-0002: not changed > > > > We have added more tries to see if the primary_slot_name becomes > active but I think it is still fragile because it is possible on slow > machines that

Re: speed up a logical replica setup

2024-05-22 Thread Euler Taveira
On Wed, May 22, 2024, at 8:19 AM, Amit Kapila wrote: > > > > v2-0001: not changed > > > > Shouldn't we modify it as per the suggestion given in the email [1]? I > am wondering if we can entirely get rid of checking the primary > business and simply rely on recovery_timeout and keep checking > serv

Re: speed up a logical replica setup

2024-05-22 Thread Amit Kapila
On Mon, May 20, 2024 at 4:30 PM Shlok Kyal wrote: > > > > I was trying to test this utility when 'sync_replication_slots' is on > > and it gets in an ERROR loop [1] and never finishes. Please find the > > postgresql.auto used on the standby attached. I think if the standby > > has enabled sync_slo

Re: speed up a logical replica setup

2024-05-20 Thread Shlok Kyal
Hi, > > I was trying to test this utility when 'sync_replication_slots' is on > and it gets in an ERROR loop [1] and never finishes. Please find the > postgresql.auto used on the standby attached. I think if the standby > has enabled sync_slots, you need to pass dbname in > GenerateRecoveryConfig()

Re: speed up a logical replica setup

2024-05-20 Thread Amit Kapila
On Sun, May 19, 2024 at 4:25 AM Thomas Munro wrote: > > 040_pg_createsubscriber.pl seems to be failing occasionally on > culicidae near "--dry-run on node S". I couldn't immediately see why. > That animal is using EXEC_BACKEND and I also saw a one-off failure a > bit like that on my own local Lin

Re: speed up a logical replica setup

2024-05-18 Thread Thomas Munro
040_pg_createsubscriber.pl seems to be failing occasionally on culicidae near "--dry-run on node S". I couldn't immediately see why. That animal is using EXEC_BACKEND and I also saw a one-off failure a bit like that on my own local Linux + EXEC_BACKEND test run (sorry I didn't keep the details aro

Re: speed up a logical replica setup

2024-04-30 Thread Amit Kapila
On Tue, Apr 30, 2024 at 12:04 PM Amit Kapila wrote: > > On Mon, Apr 29, 2024 at 5:28 PM Amit Kapila wrote: > > > > On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > > > > I was trying to test this utility when 'sync_replication_slots' is on > and it gets in an ERROR loop [1] and never finis

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Mon, Apr 29, 2024 at 5:28 PM Amit Kapila wrote: > > On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > I was trying to test this utility when 'sync_replication_slots' is on and it gets in an ERROR loop [1] and never finishes. Please find the postgresql.auto used on the standby attached. I

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > > On Mon, Apr 29, 2024, at 6:56 AM, Amit Kapila wrote: > > On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS

Re: speed up a logical replica setup

2024-04-29 Thread Euler Taveira
On Mon, Apr 29, 2024, at 6:56 AM, Amit Kapila wrote: > On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > > Why isn't recovery_timeout enough to decide

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Tue, Mar 26, 2024 at 8:24 AM Euler Taveira wrote: > > On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: > > The first patch implements a combination of (1) and (2). > > ## Analysis for failure 2 > > According to [2], the physical replication slot which is specified as > primary_

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > Why isn't recovery_timeout enough to decide if wait_for_end_recovery() > waited long enough? > > > It was an

Re: speed up a logical replica setup

2024-03-26 Thread Tomas Vondra
On 3/26/24 21:17, Euler Taveira wrote: > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: >> Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? >> Why isn't recovery_timeout enough to decide if wait_for_end_recovery() >> waited long enough? > > It was an attempt to decou

Re: speed up a logical replica setup

2024-03-26 Thread Euler Taveira
On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > Why isn't recovery_timeout enough to decide if wait_for_end_recovery() > waited long enough? It was an attempt to decoupled a connection failure (that keeps streaming

Re: speed up a logical replica setup

2024-03-26 Thread Tomas Vondra
On 3/26/24 03:53, Euler Taveira wrote: > On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: >> ## Analysis for failure 1 >> >> The failure caused by a time lag between walreceiver finishes and >> pg_is_in_recovery() >> returns true. >> >> According to the output [1], it seems that th

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Euler, > > This only drops the publications created by this tool, not the > pre-existing ones that we discussed in the link provided. Another concern around here is the case which primary subscribes changes from others. After the conversion, new subscriber also tries to connect to an

Re: speed up a logical replica setup

2024-03-25 Thread Amit Kapila
On Tue, Mar 26, 2024 at 8:27 AM Euler Taveira wrote: > > On Mon, Mar 25, 2024, at 11:33 PM, Amit Kapila wrote: > > On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > > > I have committed your version v33. I did another pass over the > > identifier and literal quoting. I added quoting f

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 11:33 PM, Amit Kapila wrote: > On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > > > I have committed your version v33. I did another pass over the > > identifier and literal quoting. I added quoting for replication slot > > names, for example, even though they

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: > ## Analysis for failure 1 > > The failure caused by a time lag between walreceiver finishes and > pg_is_in_recovery() > returns true. > > According to the output [1], it seems that the tool failed at > wait_for_end_recovery() >

Re: speed up a logical replica setup

2024-03-25 Thread Amit Kapila
On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > I have committed your version v33. I did another pass over the > identifier and literal quoting. I added quoting for replication slot > names, for example, even though they can only contain a restricted set > of characters, but it felt

Re: speed up a logical replica setup

2024-03-25 Thread vignesh C
On Mon, 25 Mar 2024 at 21:36, Hayato Kuroda (Fujitsu) wrote: > > Dear Bharath, Peter, > > > Looks like BF animals aren't happy, please check - > > > https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. > > > > Looks like sanitizer failures. There were a few messages about that > > recently,

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
Dear Bharath, Peter, > Looks like BF animals aren't happy, please check - > > https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. > > Looks like sanitizer failures. There were a few messages about that > recently, but those were all just about freeing memory after use, which > we don't ne

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 8:55 AM, Peter Eisentraut wrote: > On 22.03.24 04:31, Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > >> There is a compilation error while building postgres with the patch > >> due to a recent commit. I have attached a top-up patch v32-0003 t

Re: speed up a logical replica setup

2024-03-25 Thread Peter Eisentraut
On 25.03.24 13:36, Bharath Rupireddy wrote: On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: I have committed your version v33. Looks like BF animals aren't happy, please check - https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. Looks like sanitizer failures. There were a

Re: speed up a logical replica setup

2024-03-25 Thread Bharath Rupireddy
On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > I have committed your version v33. Looks like BF animals aren't happy, please check - https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Ser

Re: speed up a logical replica setup

2024-03-25 Thread Peter Eisentraut
On 22.03.24 04:31, Euler Taveira wrote: On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: There is a compilation error while building postgres with the patch due to a recent commit. I have attached a top-up patch v32-0003 to resolve this compilation error. I have not updated the version of the

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
> IIUC, added options were inspired by Tomas. And he said the limitation > (pub/sub/slot > name cannot be specified) was acceptable for the first version. I agree with > him. > (To be honest, I feel that options should be fewer for the first release) Just to confirm - I don't think it is not comp

RE: speed up a logical replica setup

2024-03-24 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Euler, I also want to share my opinion, just in case. > On Thu, Mar 21, 2024 at 8:00 PM Euler Taveira wrote: > > > > On Thu, Mar 21, 2024, at 10:33 AM, vignesh C wrote: > > > > If we commit this we might not be able to change the way the option > > behaves once the customers starts us

Re: speed up a logical replica setup

2024-03-22 Thread Shubham Khanna
On Fri, Mar 22, 2024 at 9:02 AM Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > > There is a compilation error while building postgres with the patch > due to a recent commit. I have attached a top-up patch v32-0003 to > resolve this compilation error. > I have not u

Re: speed up a logical replica setup

2024-03-22 Thread vignesh C
On Fri, 22 Mar 2024 at 09:01, Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > > There is a compilation error while building postgres with the patch > due to a recent commit. I have attached a top-up patch v32-0003 to > resolve this compilation error. > I have not upd

Re: speed up a logical replica setup

2024-03-21 Thread Amit Kapila
On Fri, Mar 22, 2024 at 9:44 AM Fabrízio de Royes Mello wrote: > > On Fri, Mar 22, 2024 at 12:54 AM Amit Kapila wrote: > > > > > > The user choosing to convert a physical standby to a subscriber would > > in some cases be interested in converting it for all the databases > > (say for the case of

Re: speed up a logical replica setup

2024-03-21 Thread Fabrízio de Royes Mello
On Fri, Mar 22, 2024 at 12:54 AM Amit Kapila wrote: > > > The user choosing to convert a physical standby to a subscriber would > in some cases be interested in converting it for all the databases > (say for the case of upgrade [1]) and giving options for each database > would be cumbersome for he

Re: speed up a logical replica setup

2024-03-21 Thread Amit Kapila
On Thu, Mar 21, 2024 at 8:00 PM Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 10:33 AM, vignesh C wrote: > > If we commit this we might not be able to change the way the option > behaves once the customers starts using it. How about removing these > options in the first version and adding it i

  1   2   3   4   >